diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf87d471e34a..3d93f2d032c7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,6 +32,8 @@ When changes are made to authentication and project ID-related code, authenticat Known issue: If you have installed the Google Cloud SDK, be sure to log in (using `gcloud auth login`) before running tests. Though the Datastore tests use a local Datastore emulator that doesn't require authentication, they will not run if you have the Google Cloud SDK installed but aren't authenticated. +**Please, do not use your production projects for executing integration tests.** While we do our best to make our tests independent of your project's state and content, they do perform create, modify and deletes, and you do not want to have your production data accidentally modified. + Adding Features --------------- In order to add a feature to gcloud-java: diff --git a/README.md b/README.md index 67fbc4c8e337..68c624c37489 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services. [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs) @@ -27,33 +29,33 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java:0.1.3' +compile 'com.google.gcloud:gcloud-java:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.5" ``` Example Applications -------------------- -- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html). +- [`BigQueryExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality + - Read more about using this application on the [`BigQueryExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/bigquery/BigQueryExample.html). - [`Bookshelf`](https://github.com/GoogleCloudPlatform/getting-started-java/tree/master/bookshelf) - An App Engine app that manages a virtual bookshelf. - This app uses `gcloud-java` to interface with Cloud Datastore and Cloud Storage. It also uses Cloud SQL, another Google Cloud Platform service. -- [`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) - A simple command line interface for the Cloud Datastore - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html). -- [`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html). -- [`SparkDemo`](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/managedvms/sparkjava) - An example of using gcloud-java-datastore from within the SparkJava and App Engine Managed VM frameworks. - - Read about how it works on the example's [README page](https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/managedvms/sparkjava#how-does-it-work). -- [`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality - - Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html). +- [`DatastoreExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java) - A simple command line interface for the Cloud Datastore + - Read more about using this application on the [`DatastoreExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/datastore/DatastoreExample.html). +- [`ResourceManagerExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java) - A simple command line interface providing some of Cloud Resource Manager's functionality + - Read more about using this application on the [`ResourceManagerExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/resourcemanager/ResourceManagerExample.html). +- [`SparkDemo`](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/managed_vms/sparkjava) - An example of using gcloud-java-datastore from within the SparkJava and App Engine Managed VM frameworks. + - Read about how it works on the example's [README page](https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/managed_vms/sparkjava#how-does-it-work). +- [`StorageExample`](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java) - A simple command line interface providing some of Cloud Storage's functionality + - Read more about using this application on the [`StorageExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/storage/StorageExample.html). Specifying a Project ID ----------------------- @@ -123,41 +125,39 @@ Google Cloud BigQuery (Alpha) Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Complete source code can be found at +[CreateTableAndLoadData.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java). ```java import com.google.gcloud.bigquery.BigQuery; import com.google.gcloud.bigquery.BigQueryOptions; import com.google.gcloud.bigquery.Field; -import com.google.gcloud.bigquery.JobStatus; -import com.google.gcloud.bigquery.JobInfo; -import com.google.gcloud.bigquery.LoadJobConfiguration; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; import com.google.gcloud.bigquery.Schema; import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; BigQuery bigquery = BigQueryOptions.defaultInstance().service(); TableId tableId = TableId.of("dataset", "table"); -TableInfo info = bigquery.getTable(tableId); -if (info == null) { +Table table = bigquery.getTable(tableId); +if (table == null) { System.out.println("Creating table " + tableId); Field integerField = Field.of("fieldName", Field.Type.integer()); Schema schema = Schema.of(integerField); - bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema))); + table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema))); +} +System.out.println("Loading data into table " + tableId); +Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path"); +while (!loadJob.isDone()) { + Thread.sleep(1000L); +} +if (loadJob.status().error() != null) { + System.out.println("Job completed with errors"); } else { - System.out.println("Loading data into table " + tableId); - LoadJobConfiguration configuration = LoadJobConfiguration.of(tableId, "gs://bucket/path"); - JobInfo loadJob = JobInfo.of(configuration); - loadJob = bigquery.create(loadJob); - while (loadJob.status().state() != JobStatus.State.DONE) { - Thread.sleep(1000L); - loadJob = bigquery.getJob(loadJob.jobId()); - } - if (loadJob.status().error() != null) { - System.out.println("Job completed with errors"); - } else { - System.out.println("Job succeeded"); - } + System.out.println("Job succeeded"); } ``` @@ -171,8 +171,32 @@ Google Cloud Datastore #### Preview -Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Here are two code snippets showing simple usage examples from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. + +The first snippet shows how to create a Datastore entity. Complete source code can be found at +[CreateEntity.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java). + +```java +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; +Datastore datastore = DatastoreOptions.defaultInstance().service(); +KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); +Key key = keyFactory.newKey("keyName"); +Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 30) + .set("access_time", DateTime.now()) + .build(); +datastore.put(entity); +``` +The second snippet shows how to update a Datastore entity if it exists. Complete source code can be +found at +[UpdateEntity.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java). ```java import com.google.gcloud.datastore.Datastore; import com.google.gcloud.datastore.DatastoreOptions; @@ -182,17 +206,10 @@ import com.google.gcloud.datastore.Key; import com.google.gcloud.datastore.KeyFactory; Datastore datastore = DatastoreOptions.defaultInstance().service(); -KeyFactory keyFactory = datastore.newKeyFactory().kind(KIND); -Key key = keyFactory.newKey(keyName); +KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); +Key key = keyFactory.newKey("keyName"); Entity entity = datastore.get(key); -if (entity == null) { - entity = Entity.builder(key) - .set("name", "John Do") - .set("age", 30) - .set("access_time", DateTime.now()) - .build(); - datastore.put(entity); -} else { +if (entity != null) { System.out.println("Updating access_time for " + entity.getString("name")); entity = Entity.builder(entity) .set("access_time", DateTime.now()) @@ -210,7 +227,8 @@ Google Cloud Resource Manager (Alpha) #### Preview Here is a code snippet showing a simple usage example. Note that you must supply Google SDK credentials for this service, not other forms of authentication listed in the [Authentication section](#authentication). - +Complete source code can be found at +[UpdateAndListProjects.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java). ```java import com.google.gcloud.resourcemanager.Project; import com.google.gcloud.resourcemanager.ResourceManager; @@ -219,14 +237,15 @@ import com.google.gcloud.resourcemanager.ResourceManagerOptions; import java.util.Iterator; ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); -Project myProject = resourceManager.get("some-project-id"); // Use an existing project's ID -Project newProject = myProject.toBuilder() - .addLabel("launch-status", "in-development") - .build() - .replace(); -System.out.println("Updated the labels of project " + newProject.projectId() - + " to be " + newProject.labels()); -// List all the projects you have permission to view. +Project project = resourceManager.get("some-project-id"); // Use an existing project's ID +if (project != null) { + Project newProject = project.toBuilder() + .addLabel("launch-status", "in-development") + .build() + .replace(); + System.out.println("Updated the labels of project " + newProject.projectId() + + " to be " + newProject.labels()); +} Iterator projectIterator = resourceManager.list().iterateAll(); System.out.println("Projects I can view:"); while (projectIterator.hasNext()) { @@ -244,8 +263,28 @@ Google Cloud Storage #### Preview -Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. +Here are two code snippets showing simple usage examples from within Compute/App Engine. Note that you must [supply credentials](#authentication) and a project ID if running this snippet elsewhere. + +The first snippet shows how to create a Storage blob. Complete source code can be found at +[CreateBlob.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java). + +```java +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.StorageOptions; +Storage storage = StorageOptions.defaultInstance().service(); +BlobId blobId = BlobId.of("bucket", "blob_name"); +BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build(); +Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8)); +``` +The second snippet shows how to update a Storage blob if it exists. Complete source code can be +found at +[UpdateBlob.java](./gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java). ```java import static java.nio.charset.StandardCharsets.UTF_8; @@ -259,12 +298,8 @@ import java.nio.channels.WritableByteChannel; Storage storage = StorageOptions.defaultInstance().service(); BlobId blobId = BlobId.of("bucket", "blob_name"); -Blob blob = Blob.get(storage, blobId); -if (blob == null) { - BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build(); - storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8)); -} else { - System.out.println("Updating content for " + blobId.name()); +Blob blob = storage.get(blobId); +if (blob != null) { byte[] prevContent = blob.content(); System.out.println(new String(prevContent, UTF_8)); WritableByteChannel channel = blob.writer(); diff --git a/codacy-conf.json b/codacy-conf.json new file mode 100644 index 000000000000..e8c819684c9c --- /dev/null +++ b/codacy-conf.json @@ -0,0 +1 @@ +{"patterns":[{"patternId":"Custom_Javascript_Scopes","enabled":true},{"patternId":"Custom_Javascript_EvalWith","enabled":true},{"patternId":"Custom_Javascript_TryCatch","enabled":true},{"patternId":"Custom_Scala_NonFatal","enabled":true},{"patternId":"bitwise","enabled":true},{"patternId":"maxparams","enabled":true},{"patternId":"CSSLint_universal_selector","enabled":true},{"patternId":"CSSLint_unqualified_attributes","enabled":true},{"patternId":"CSSLint_zero_units","enabled":true},{"patternId":"CSSLint_overqualified_elements","enabled":true},{"patternId":"CSSLint_shorthand","enabled":true},{"patternId":"CSSLint_duplicate_background_images","enabled":true},{"patternId":"CSSLint_box_model","enabled":true},{"patternId":"CSSLint_compatible_vendor_prefixes","enabled":true},{"patternId":"CSSLint_display_property_grouping","enabled":true},{"patternId":"CSSLint_duplicate_properties","enabled":true},{"patternId":"CSSLint_empty_rules","enabled":true},{"patternId":"CSSLint_errors","enabled":true},{"patternId":"CSSLint_gradients","enabled":true},{"patternId":"CSSLint_important","enabled":true},{"patternId":"CSSLint_known_properties","enabled":true},{"patternId":"CSSLint_text_indent","enabled":true},{"patternId":"CSSLint_unique_headings","enabled":true},{"patternId":"PyLint_E0100","enabled":true},{"patternId":"PyLint_E0101","enabled":true},{"patternId":"PyLint_E0102","enabled":true},{"patternId":"PyLint_E0103","enabled":true},{"patternId":"PyLint_E0104","enabled":true},{"patternId":"PyLint_E0105","enabled":true},{"patternId":"PyLint_E0106","enabled":true},{"patternId":"PyLint_E0107","enabled":true},{"patternId":"PyLint_E0108","enabled":true},{"patternId":"PyLint_E0202","enabled":true},{"patternId":"PyLint_E0203","enabled":true},{"patternId":"PyLint_E0211","enabled":true},{"patternId":"PyLint_E0601","enabled":true},{"patternId":"PyLint_E0603","enabled":true},{"patternId":"PyLint_E0604","enabled":true},{"patternId":"PyLint_E0701","enabled":true},{"patternId":"PyLint_E0702","enabled":true},{"patternId":"PyLint_E0710","enabled":true},{"patternId":"PyLint_E0711","enabled":true},{"patternId":"PyLint_E0712","enabled":true},{"patternId":"PyLint_E1003","enabled":true},{"patternId":"PyLint_E1102","enabled":true},{"patternId":"PyLint_E1111","enabled":true},{"patternId":"PyLint_E1120","enabled":true},{"patternId":"PyLint_E1121","enabled":true},{"patternId":"PyLint_E1123","enabled":true},{"patternId":"PyLint_E1124","enabled":true},{"patternId":"PyLint_E1200","enabled":true},{"patternId":"PyLint_E1201","enabled":true},{"patternId":"PyLint_E1205","enabled":true},{"patternId":"PyLint_E1206","enabled":true},{"patternId":"PyLint_E1300","enabled":true},{"patternId":"PyLint_E1301","enabled":true},{"patternId":"PyLint_E1302","enabled":true},{"patternId":"PyLint_E1303","enabled":true},{"patternId":"PyLint_E1304","enabled":true},{"patternId":"PyLint_E1305","enabled":true},{"patternId":"PyLint_E1306","enabled":true},{"patternId":"rulesets-codesize.xml-CyclomaticComplexity","enabled":true},{"patternId":"rulesets-codesize.xml-NPathComplexity","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveMethodLength","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveClassLength","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveParameterList","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessivePublicCount","enabled":true},{"patternId":"rulesets-codesize.xml-TooManyFields","enabled":true},{"patternId":"rulesets-codesize.xml-TooManyMethods","enabled":true},{"patternId":"rulesets-codesize.xml-ExcessiveClassComplexity","enabled":true},{"patternId":"rulesets-controversial.xml-Superglobals","enabled":true},{"patternId":"rulesets-design.xml-ExitExpression","enabled":true},{"patternId":"rulesets-design.xml-EvalExpression","enabled":true},{"patternId":"rulesets-design.xml-GotoStatement","enabled":true},{"patternId":"rulesets-design.xml-NumberOfChildren","enabled":true},{"patternId":"rulesets-design.xml-DepthOfInheritance","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedPrivateField","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedLocalVariable","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedPrivateMethod","enabled":true},{"patternId":"rulesets-unusedcode.xml-UnusedFormalParameter","enabled":true},{"patternId":"PyLint_C0303","enabled":true},{"patternId":"PyLint_C1001","enabled":true},{"patternId":"rulesets-naming.xml-ShortVariable","enabled":true},{"patternId":"rulesets-naming.xml-LongVariable","enabled":true},{"patternId":"rulesets-naming.xml-ShortMethodName","enabled":true},{"patternId":"rulesets-naming.xml-ConstantNamingConventions","enabled":true},{"patternId":"rulesets-naming.xml-BooleanGetMethodName","enabled":true},{"patternId":"PyLint_W0101","enabled":true},{"patternId":"PyLint_W0102","enabled":true},{"patternId":"PyLint_W0104","enabled":true},{"patternId":"PyLint_W0105","enabled":true},{"patternId":"Custom_Scala_GetCalls","enabled":true},{"patternId":"ScalaStyle_EqualsHashCodeChecker","enabled":true},{"patternId":"ScalaStyle_ParameterNumberChecker","enabled":true},{"patternId":"ScalaStyle_ReturnChecker","enabled":true},{"patternId":"ScalaStyle_NullChecker","enabled":true},{"patternId":"ScalaStyle_NoCloneChecker","enabled":true},{"patternId":"ScalaStyle_NoFinalizeChecker","enabled":true},{"patternId":"ScalaStyle_CovariantEqualsChecker","enabled":true},{"patternId":"ScalaStyle_StructuralTypeChecker","enabled":true},{"patternId":"ScalaStyle_MethodLengthChecker","enabled":true},{"patternId":"ScalaStyle_NumberOfMethodsInTypeChecker","enabled":true},{"patternId":"ScalaStyle_WhileChecker","enabled":true},{"patternId":"ScalaStyle_VarFieldChecker","enabled":true},{"patternId":"ScalaStyle_VarLocalChecker","enabled":true},{"patternId":"ScalaStyle_RedundantIfChecker","enabled":true},{"patternId":"ScalaStyle_DeprecatedJavaChecker","enabled":true},{"patternId":"ScalaStyle_EmptyClassChecker","enabled":true},{"patternId":"ScalaStyle_NotImplementedErrorUsage","enabled":true},{"patternId":"Custom_Scala_GroupImports","enabled":true},{"patternId":"Custom_Scala_ReservedKeywords","enabled":true},{"patternId":"Custom_Scala_ElseIf","enabled":true},{"patternId":"Custom_Scala_CallByNameAsLastArguments","enabled":true},{"patternId":"Custom_Scala_WildcardImportOnMany","enabled":true},{"patternId":"Custom_Scala_UtilTryForTryCatch","enabled":true},{"patternId":"Custom_Scala_ProhibitObjectName","enabled":true},{"patternId":"Custom_Scala_ImportsAtBeginningOfPackage","enabled":true},{"patternId":"Custom_Scala_NameResultsAndParameters","enabled":true},{"patternId":"Custom_Scala_IncompletePatternMatching","enabled":true},{"patternId":"Custom_Scala_UsefulTypeAlias","enabled":true},{"patternId":"Custom_Scala_JavaThreads","enabled":true},{"patternId":"Custom_Scala_DirectPromiseCreation","enabled":true},{"patternId":"Custom_Scala_StructuralTypes","enabled":true},{"patternId":"Custom_Scala_CollectionLastHead","enabled":true},{"patternId":"PyLint_W0106","enabled":true},{"patternId":"PyLint_W0107","enabled":true},{"patternId":"PyLint_W0108","enabled":true},{"patternId":"PyLint_W0109","enabled":true},{"patternId":"PyLint_W0110","enabled":true},{"patternId":"PyLint_W0120","enabled":true},{"patternId":"PyLint_W0122","enabled":true},{"patternId":"PyLint_W0150","enabled":true},{"patternId":"PyLint_W0199","enabled":true},{"patternId":"rulesets-cleancode.xml-ElseExpression","enabled":true},{"patternId":"rulesets-cleancode.xml-StaticAccess","enabled":true},{"patternId":"ScalaStyle_NonASCIICharacterChecker","enabled":true},{"patternId":"ScalaStyle_FieldNamesChecker","enabled":true},{"patternId":"Custom_Scala_WithNameCalls","enabled":true},{"patternId":"strictexception_AvoidRethrowingException","enabled":true},{"patternId":"strings_AppendCharacterWithChar","enabled":true},{"patternId":"braces_IfElseStmtsMustUseBraces","enabled":true},{"patternId":"basic_AvoidDecimalLiteralsInBigDecimalConstructor","enabled":true},{"patternId":"basic_CheckSkipResult","enabled":true},{"patternId":"javabeans_MissingSerialVersionUID","enabled":true},{"patternId":"migrating_ShortInstantiation","enabled":true},{"patternId":"design_AvoidInstanceofChecksInCatchClause","enabled":true},{"patternId":"naming_LongVariable","enabled":true},{"patternId":"migrating_ReplaceEnumerationWithIterator","enabled":true},{"patternId":"j2ee_DoNotCallSystemExit","enabled":true},{"patternId":"unusedcode_UnusedLocalVariable","enabled":true},{"patternId":"strings_InefficientStringBuffering","enabled":true},{"patternId":"basic_DontUseFloatTypeForLoopIndices","enabled":true},{"patternId":"basic_AvoidBranchingStatementAsLastInLoop","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseTestAnnotation","enabled":true},{"patternId":"optimizations_AddEmptyString","enabled":true},{"patternId":"logging-jakarta-commons_ProperLogger","enabled":true},{"patternId":"optimizations_RedundantFieldInitializer","enabled":true},{"patternId":"logging-java_AvoidPrintStackTrace","enabled":true},{"patternId":"empty_EmptyFinallyBlock","enabled":true},{"patternId":"design_CompareObjectsWithEquals","enabled":true},{"patternId":"basic_ClassCastExceptionWithToArray","enabled":true},{"patternId":"strictexception_DoNotExtendJavaLangError","enabled":true},{"patternId":"junit_UnnecessaryBooleanAssertion","enabled":true},{"patternId":"design_SimplifyBooleanExpressions","enabled":true},{"patternId":"basic_ForLoopShouldBeWhileLoop","enabled":true},{"patternId":"basic_BigIntegerInstantiation","enabled":true},{"patternId":"optimizations_UseArrayListInsteadOfVector","enabled":true},{"patternId":"optimizations_UnnecessaryWrapperObjectCreation","enabled":true},{"patternId":"strings_StringBufferInstantiationWithChar","enabled":true},{"patternId":"basic_JumbledIncrementer","enabled":true},{"patternId":"design_SwitchStmtsShouldHaveDefault","enabled":true},{"patternId":"strictexception_AvoidThrowingRawExceptionTypes","enabled":true},{"patternId":"migrating_LongInstantiation","enabled":true},{"patternId":"design_SimplifyBooleanReturns","enabled":true},{"patternId":"empty_EmptyInitializer","enabled":true},{"patternId":"design_FieldDeclarationsShouldBeAtStartOfClass","enabled":true},{"patternId":"unnecessary_UnnecessaryConversionTemporary","enabled":true},{"patternId":"design_AvoidProtectedFieldInFinalClass","enabled":true},{"patternId":"junit_UseAssertTrueInsteadOfAssertEquals","enabled":true},{"patternId":"naming_PackageCase","enabled":true},{"patternId":"migrating_JUnitUseExpected","enabled":true},{"patternId":"controversial_UnnecessaryConstructor","enabled":true},{"patternId":"naming_MethodNamingConventions","enabled":true},{"patternId":"design_DefaultLabelNotLastInSwitchStmt","enabled":true},{"patternId":"basic_UnconditionalIfStatement","enabled":true},{"patternId":"design_SingularField","enabled":true},{"patternId":"design_AssignmentToNonFinalStatic","enabled":true},{"patternId":"braces_WhileLoopsMustUseBraces","enabled":true},{"patternId":"logging-java_SystemPrintln","enabled":true},{"patternId":"strings_UseStringBufferLength","enabled":true},{"patternId":"controversial_AvoidUsingNativeCode","enabled":true},{"patternId":"strictexception_AvoidLosingExceptionInformation","enabled":true},{"patternId":"imports_ImportFromSamePackage","enabled":true},{"patternId":"finalizers_AvoidCallingFinalize","enabled":true},{"patternId":"finalizers_FinalizeOverloaded","enabled":true},{"patternId":"naming_ClassNamingConventions","enabled":true},{"patternId":"logging-java_LoggerIsNotStaticFinal","enabled":true},{"patternId":"finalizers_FinalizeOnlyCallsSuperFinalize","enabled":true},{"patternId":"unnecessary_UselessOverridingMethod","enabled":true},{"patternId":"naming_SuspiciousConstantFieldName","enabled":true},{"patternId":"design_OptimizableToArrayCall","enabled":true},{"patternId":"imports_UnnecessaryFullyQualifiedName","enabled":true},{"patternId":"migrating_ReplaceHashtableWithMap","enabled":true},{"patternId":"unusedcode_UnusedPrivateField","enabled":true},{"patternId":"strings_UnnecessaryCaseChange","enabled":true},{"patternId":"migrating_IntegerInstantiation","enabled":true},{"patternId":"design_NonStaticInitializer","enabled":true},{"patternId":"design_MissingBreakInSwitch","enabled":true},{"patternId":"design_AvoidReassigningParameters","enabled":true},{"patternId":"basic_AvoidThreadGroup","enabled":true},{"patternId":"empty_EmptyCatchBlock","parameters":{"allowCommentedBlocks":"true"},"enabled":true},{"patternId":"codesize_ExcessiveParameterList","parameters":{"minimum":"8","violationSuppressRegex":"\"\"","violationSuppressXPath":"\"\""},"enabled":true},{"patternId":"naming_SuspiciousHashcodeMethodName","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseBeforeAnnotation","enabled":true},{"patternId":"design_UncommentedEmptyMethodBody","enabled":true},{"patternId":"basic_BrokenNullCheck","enabled":true},{"patternId":"strings_ConsecutiveLiteralAppends","enabled":true},{"patternId":"strings_StringInstantiation","enabled":true},{"patternId":"design_EqualsNull","enabled":true},{"patternId":"basic_OverrideBothEqualsAndHashcode","enabled":true},{"patternId":"design_InstantiationToGetClass","enabled":true},{"patternId":"basic_BooleanInstantiation","enabled":true},{"patternId":"strings_AvoidStringBufferField","enabled":true},{"patternId":"basic_ReturnFromFinallyBlock","enabled":true},{"patternId":"empty_EmptyTryBlock","enabled":true},{"patternId":"naming_SuspiciousEqualsMethodName","enabled":true},{"patternId":"basic_ExtendsObject","enabled":true},{"patternId":"strings_UselessStringValueOf","enabled":true},{"patternId":"design_UnsynchronizedStaticDateFormatter","enabled":true},{"patternId":"design_UseCollectionIsEmpty","enabled":true},{"patternId":"controversial_AvoidFinalLocalVariable","enabled":true},{"patternId":"strictexception_AvoidThrowingNullPointerException","enabled":true},{"patternId":"design_AvoidProtectedMethodInFinalClassNotExtending","enabled":true},{"patternId":"optimizations_PrematureDeclaration","enabled":true},{"patternId":"empty_EmptySwitchStatements","enabled":true},{"patternId":"basic_MisplacedNullCheck","enabled":true},{"patternId":"optimizations_UseStringBufferForStringAppends","enabled":true},{"patternId":"strings_StringToString","enabled":true},{"patternId":"naming_MethodWithSameNameAsEnclosingClass","enabled":true},{"patternId":"migrating_ReplaceVectorWithList","enabled":true},{"patternId":"imports_UnusedImports","enabled":true},{"patternId":"unnecessary_UnnecessaryFinalModifier","enabled":true},{"patternId":"basic_AvoidMultipleUnaryOperators","enabled":true},{"patternId":"junit_SimplifyBooleanAssertion","enabled":true},{"patternId":"unnecessary_UselessParentheses","enabled":true},{"patternId":"design_IdempotentOperations","enabled":true},{"patternId":"braces_IfStmtsMustUseBraces","enabled":true},{"patternId":"strings_UseIndexOfChar","enabled":true},{"patternId":"naming_NoPackage","enabled":true},{"patternId":"finalizers_FinalizeDoesNotCallSuperFinalize","enabled":true},{"patternId":"design_UseVarargs","enabled":true},{"patternId":"unusedcode_UnusedFormalParameter","enabled":true},{"patternId":"design_ReturnEmptyArrayRatherThanNull","enabled":true},{"patternId":"junit_UseAssertNullInsteadOfAssertTrue","enabled":true},{"patternId":"design_UseUtilityClass","enabled":true},{"patternId":"design_AvoidDeeplyNestedIfStmts","enabled":true},{"patternId":"empty_EmptyStatementNotInLoop","enabled":true},{"patternId":"junit_UseAssertSameInsteadOfAssertTrue","enabled":true},{"patternId":"braces_ForLoopsMustUseBraces","enabled":true},{"patternId":"controversial_DoNotCallGarbageCollectionExplicitly","enabled":true},{"patternId":"naming_GenericsNaming","enabled":true},{"patternId":"strings_UseEqualsToCompareStrings","enabled":true},{"patternId":"optimizations_AvoidArrayLoops","enabled":true},{"patternId":"empty_EmptyStaticInitializer","enabled":true},{"patternId":"design_UncommentedEmptyConstructor","enabled":true},{"patternId":"empty_EmptyStatementBlock","enabled":true},{"patternId":"basic_CollapsibleIfStatements","enabled":true},{"patternId":"design_FinalFieldCouldBeStatic","enabled":true},{"patternId":"logging-java_MoreThanOneLogger","enabled":true},{"patternId":"codesize_ExcessiveClassLength","enabled":true},{"patternId":"design_ImmutableField","enabled":true},{"patternId":"controversial_OneDeclarationPerLine","enabled":true},{"patternId":"empty_EmptyWhileStmt","enabled":true},{"patternId":"unnecessary_UnnecessaryReturn","enabled":true},{"patternId":"strings_InefficientEmptyStringCheck","enabled":true},{"patternId":"design_UseNotifyAllInsteadOfNotify","enabled":true},{"patternId":"strictexception_DoNotThrowExceptionInFinally","enabled":true},{"patternId":"junit_UseAssertEqualsInsteadOfAssertTrue","enabled":true},{"patternId":"typeresolution_CloneMethodMustImplementCloneable","enabled":true},{"patternId":"codesize_NPathComplexity","enabled":true},{"patternId":"imports_DontImportJavaLang","enabled":true},{"patternId":"empty_EmptySynchronizedBlock","enabled":true},{"patternId":"migrating_JUnit4TestShouldUseAfterAnnotation","enabled":true},{"patternId":"design_AvoidConstantsInterface","enabled":true},{"patternId":"unnecessary_UselessOperationOnImmutable","enabled":true},{"patternId":"design_PositionLiteralsFirstInComparisons","enabled":true},{"patternId":"migrating_ByteInstantiation","enabled":true},{"patternId":"junit_JUnitSpelling","enabled":true},{"patternId":"junit_JUnitTestsShouldIncludeAssert","enabled":true},{"patternId":"finalizers_EmptyFinalizer","enabled":true},{"patternId":"design_NonCaseLabelInSwitchStatement","enabled":true},{"patternId":"android_DoNotHardCodeSDCard","enabled":true},{"patternId":"design_LogicInversion","enabled":true},{"patternId":"unusedcode_UnusedPrivateMethod","enabled":true},{"patternId":"naming_AvoidDollarSigns","enabled":true},{"patternId":"finalizers_FinalizeShouldBeProtected","enabled":true},{"patternId":"clone_ProperCloneImplementation","enabled":true},{"patternId":"basic_CheckResultSet","enabled":true},{"patternId":"controversial_AvoidPrefixingMethodParameters","enabled":true},{"patternId":"migrating_JUnit4SuitesShouldUseSuiteAnnotation","enabled":true},{"patternId":"empty_EmptyIfStmt","enabled":true},{"patternId":"basic_DontCallThreadRun","enabled":true},{"patternId":"junit_JUnitStaticSuite","enabled":true},{"patternId":"optimizations_UseArraysAsList","enabled":true},{"patternId":"design_MissingStaticMethodInNonInstantiatableClass","enabled":true},{"patternId":"unusedcode_UnusedModifier","enabled":true},{"patternId":"Style_MethodName","enabled":true},{"patternId":"Metrics_CyclomaticComplexity","enabled":true},{"patternId":"Lint_DuplicateMethods","enabled":true},{"patternId":"Style_Lambda","enabled":true},{"patternId":"Lint_UselessSetterCall","enabled":true},{"patternId":"Style_VariableName","enabled":true},{"patternId":"Lint_AmbiguousOperator","enabled":true},{"patternId":"Style_LeadingCommentSpace","enabled":true},{"patternId":"Style_CaseEquality","enabled":true},{"patternId":"Lint_StringConversionInInterpolation","enabled":true},{"patternId":"Performance_ReverseEach","enabled":true},{"patternId":"Lint_LiteralInCondition","enabled":true},{"patternId":"Performance_Sample","enabled":true},{"patternId":"Style_NonNilCheck","enabled":true},{"patternId":"Lint_RescueException","enabled":true},{"patternId":"Lint_UselessElseWithoutRescue","enabled":true},{"patternId":"Style_ConstantName","enabled":true},{"patternId":"Lint_LiteralInInterpolation","enabled":true},{"patternId":"Lint_NestedMethodDefinition","enabled":true},{"patternId":"Style_DoubleNegation","enabled":true},{"patternId":"Lint_SpaceBeforeFirstArg","enabled":true},{"patternId":"Lint_Debugger","enabled":true},{"patternId":"Style_ClassVars","enabled":true},{"patternId":"Lint_EmptyEnsure","enabled":true},{"patternId":"Style_MultilineBlockLayout","enabled":true},{"patternId":"Lint_UnusedBlockArgument","enabled":true},{"patternId":"Lint_UselessAccessModifier","enabled":true},{"patternId":"Performance_Size","enabled":true},{"patternId":"Lint_EachWithObjectArgument","enabled":true},{"patternId":"Style_Alias","enabled":true},{"patternId":"Lint_Loop","enabled":true},{"patternId":"Style_NegatedWhile","enabled":true},{"patternId":"Style_ColonMethodCall","enabled":true},{"patternId":"Lint_AmbiguousRegexpLiteral","enabled":true},{"patternId":"Lint_UnusedMethodArgument","enabled":true},{"patternId":"Style_MultilineIfThen","enabled":true},{"patternId":"Lint_EnsureReturn","enabled":true},{"patternId":"Style_NegatedIf","enabled":true},{"patternId":"Lint_Eval","enabled":true},{"patternId":"Style_NilComparison","enabled":true},{"patternId":"Style_ArrayJoin","enabled":true},{"patternId":"Lint_ConditionPosition","enabled":true},{"patternId":"Lint_UnreachableCode","enabled":true},{"patternId":"Performance_Count","enabled":true},{"patternId":"Lint_EmptyInterpolation","enabled":true},{"patternId":"Style_LambdaCall","enabled":true},{"patternId":"Lint_HandleExceptions","enabled":true},{"patternId":"Lint_ShadowingOuterLocalVariable","enabled":true},{"patternId":"Lint_EndAlignment","enabled":true},{"patternId":"Style_MultilineTernaryOperator","enabled":true},{"patternId":"Style_AutoResourceCleanup","enabled":true},{"patternId":"Lint_ElseLayout","enabled":true},{"patternId":"Style_NestedTernaryOperator","enabled":true},{"patternId":"Style_OneLineConditional","enabled":true},{"patternId":"Style_EmptyElse","enabled":true},{"patternId":"Lint_UselessComparison","enabled":true},{"patternId":"Metrics_PerceivedComplexity","enabled":true},{"patternId":"Style_InfiniteLoop","enabled":true},{"patternId":"Rails_Date","enabled":true},{"patternId":"Style_EvenOdd","enabled":true},{"patternId":"Style_IndentationConsistency","enabled":true},{"patternId":"Style_ModuleFunction","enabled":true},{"patternId":"Lint_UselessAssignment","enabled":true},{"patternId":"Style_EachWithObject","enabled":true},{"patternId":"Performance_Detect","enabled":true},{"patternId":"duplicate_key","enabled":true},{"patternId":"no_interpolation_in_single_quotes","enabled":true},{"patternId":"no_backticks","enabled":true},{"patternId":"no_unnecessary_fat_arrows","enabled":true},{"patternId":"indentation","enabled":true},{"patternId":"ensure_comprehensions","enabled":true},{"patternId":"no_stand_alone_at","enabled":true},{"patternId":"cyclomatic_complexity","enabled":true},{"patternId":"Deserialize","enabled":true},{"patternId":"SymbolDoS","enabled":true},{"patternId":"SkipBeforeFilter","enabled":true},{"patternId":"SanitizeMethods","enabled":true},{"patternId":"SelectTag","enabled":true},{"patternId":"XMLDoS","enabled":true},{"patternId":"SimpleFormat","enabled":true},{"patternId":"Evaluation","enabled":true},{"patternId":"BasicAuth","enabled":true},{"patternId":"JRubyXML","enabled":true},{"patternId":"RenderInline","enabled":true},{"patternId":"YAMLParsing","enabled":true},{"patternId":"Redirect","enabled":true},{"patternId":"UnsafeReflection","enabled":true},{"patternId":"SSLVerify","enabled":true},{"patternId":"HeaderDoS","enabled":true},{"patternId":"TranslateBug","enabled":true},{"patternId":"Execute","enabled":true},{"patternId":"JSONParsing","enabled":true},{"patternId":"LinkTo","enabled":true},{"patternId":"FileDisclosure","enabled":true},{"patternId":"SafeBufferManipulation","enabled":true},{"patternId":"ModelAttributes","enabled":true},{"patternId":"ResponseSplitting","enabled":true},{"patternId":"DigestDoS","enabled":true},{"patternId":"Send","enabled":true},{"patternId":"MailTo","enabled":true},{"patternId":"SymbolDoSCVE","enabled":true},{"patternId":"StripTags","enabled":true},{"patternId":"MassAssignment","enabled":true},{"patternId":"RegexDoS","enabled":true},{"patternId":"SelectVulnerability","enabled":true},{"patternId":"FileAccess","enabled":true},{"patternId":"ContentTag","enabled":true},{"patternId":"SessionSettings","enabled":true},{"patternId":"FilterSkipping","enabled":true},{"patternId":"CreateWith","enabled":true},{"patternId":"JSONEncoding","enabled":true},{"patternId":"SQLCVEs","enabled":true},{"patternId":"ForgerySetting","enabled":true},{"patternId":"QuoteTableName","enabled":true},{"patternId":"I18nXSS","enabled":true},{"patternId":"WithoutProtection","enabled":true},{"patternId":"CrossSiteScripting","enabled":true},{"patternId":"SingleQuotes","enabled":true},{"patternId":"NestedAttributes","enabled":true},{"patternId":"DetailedExceptions","enabled":true},{"patternId":"LinkToHref","enabled":true},{"patternId":"RenderDoS","enabled":true},{"patternId":"ModelSerialize","enabled":true},{"patternId":"SQL","enabled":true},{"patternId":"Render","enabled":true},{"patternId":"UnscopedFind","enabled":true},{"patternId":"ValidationRegex","enabled":true},{"patternId":"EscapeFunction","enabled":true},{"patternId":"Custom_Scala_FieldNamesChecker","enabled":true},{"patternId":"Custom_Scala_ObjDeserialization","enabled":true},{"patternId":"Custom_Scala_RSAPadding","enabled":true},{"patternId":"ESLint_no-extra-boolean-cast","enabled":true},{"patternId":"ESLint_no-iterator","enabled":true},{"patternId":"ESLint_no-invalid-regexp","enabled":true},{"patternId":"ESLint_no-obj-calls","enabled":true},{"patternId":"ESLint_no-sparse-arrays","enabled":true},{"patternId":"ESLint_no-unreachable","enabled":true},{"patternId":"ESLint_no-dupe-keys","enabled":true},{"patternId":"ESLint_no-multi-str","enabled":true},{"patternId":"ESLint_no-extend-native","enabled":true},{"patternId":"ESLint_guard-for-in","enabled":true},{"patternId":"ESLint_no-func-assign","enabled":true},{"patternId":"ESLint_no-extra-semi","enabled":true},{"patternId":"ESLint_camelcase","enabled":true},{"patternId":"ESLint_no-mixed-spaces-and-tabs","enabled":true},{"patternId":"ESLint_no-undef","enabled":true},{"patternId":"ESLint_semi","enabled":true},{"patternId":"ESLint_no-empty-character-class","enabled":true},{"patternId":"ESLint_complexity","enabled":true},{"patternId":"ESLint_no-dupe-class-members","enabled":true},{"patternId":"ESLint_no-debugger","enabled":true},{"patternId":"ESLint_block-scoped-var","enabled":true},{"patternId":"ESLint_no-loop-func","enabled":true},{"patternId":"ESLint_no-use-before-define","enabled":true},{"patternId":"ESLint_no-console","enabled":true},{"patternId":"ESLint_require-yield","enabled":true},{"patternId":"ESLint_no-redeclare","enabled":true},{"patternId":"ESLint_no-undefined","enabled":true},{"patternId":"ESLint_use-isnan","enabled":true},{"patternId":"ESLint_no-control-regex","enabled":true},{"patternId":"ESLint_no-const-assign","enabled":true},{"patternId":"ESLint_no-new","enabled":true},{"patternId":"ESLint_new-cap","enabled":true},{"patternId":"ESLint_no-irregular-whitespace","enabled":true},{"patternId":"ESLint_object-shorthand","enabled":true},{"patternId":"ESLint_no-ex-assign","enabled":true},{"patternId":"ESLint_wrap-iife","enabled":true},{"patternId":"ESLint_arrow-parens","enabled":true},{"patternId":"ESLint_no-constant-condition","enabled":true},{"patternId":"ESLint_no-octal","enabled":true},{"patternId":"ESLint_no-dupe-args","enabled":true},{"patternId":"ESLint_quotes","enabled":true},{"patternId":"ESLint_no-fallthrough","enabled":true},{"patternId":"ESLint_no-delete-var","enabled":true},{"patternId":"ESLint_no-caller","enabled":true},{"patternId":"ESLint_no-cond-assign","enabled":true},{"patternId":"ESLint_no-this-before-super","enabled":true},{"patternId":"ESLint_no-negated-in-lhs","enabled":true},{"patternId":"ESLint_no-inner-declarations","enabled":true},{"patternId":"ESLint_eqeqeq","enabled":true},{"patternId":"ESLint_curly","enabled":true},{"patternId":"ESLint_arrow-spacing","enabled":true},{"patternId":"ESLint_no-empty","enabled":true},{"patternId":"ESLint_no-unused-vars","enabled":true},{"patternId":"ESLint_generator-star-spacing","enabled":true},{"patternId":"ESLint_no-duplicate-case","enabled":true},{"patternId":"ESLint_valid-typeof","enabled":true},{"patternId":"ESLint_no-regex-spaces","enabled":true},{"patternId":"ESLint_no-class-assign","enabled":true},{"patternId":"PyLint_W0221","enabled":true},{"patternId":"PyLint_E0117","enabled":true},{"patternId":"PyLint_E0001","enabled":true},{"patternId":"PyLint_E0241","enabled":true},{"patternId":"PyLint_W0404","enabled":true},{"patternId":"PyLint_E0704","enabled":true},{"patternId":"PyLint_E0703","enabled":true},{"patternId":"PyLint_E0302","enabled":true},{"patternId":"PyLint_W1301","enabled":true},{"patternId":"PyLint_R0201","enabled":true},{"patternId":"PyLint_E0113","enabled":true},{"patternId":"PyLint_W0410","enabled":true},{"patternId":"PyLint_C0123","enabled":true},{"patternId":"PyLint_E0115","enabled":true},{"patternId":"PyLint_E0114","enabled":true},{"patternId":"PyLint_E1126","enabled":true},{"patternId":"PyLint_W0702","enabled":true},{"patternId":"PyLint_W1303","enabled":true},{"patternId":"PyLint_W0622","enabled":true},{"patternId":"PyLint_W0222","enabled":true},{"patternId":"PyLint_W0233","enabled":true},{"patternId":"PyLint_W1305","enabled":true},{"patternId":"PyLint_E1127","enabled":true},{"patternId":"PyLint_E0112","enabled":true},{"patternId":"PyLint_W0611","enabled":true},{"patternId":"PyLint_W0601","enabled":true},{"patternId":"PyLint_W1300","enabled":true},{"patternId":"PyLint_W0124","enabled":true},{"patternId":"PyLint_R0203","enabled":true},{"patternId":"PyLint_E0236","enabled":true},{"patternId":"PyLint_W0612","enabled":true},{"patternId":"PyLint_W0604","enabled":true},{"patternId":"PyLint_W0705","enabled":true},{"patternId":"PyLint_E0238","enabled":true},{"patternId":"PyLint_W0602","enabled":true},{"patternId":"PyLint_R0102","enabled":true},{"patternId":"PyLint_R0202","enabled":true},{"patternId":"PyLint_E0240","enabled":true},{"patternId":"PyLint_W0623","enabled":true},{"patternId":"PyLint_W0711","enabled":true},{"patternId":"PyLint_E0116","enabled":true},{"patternId":"PyLint_E0239","enabled":true},{"patternId":"PyLint_E1132","enabled":true},{"patternId":"PyLint_W1307","enabled":true},{"patternId":"PyLint_C0200","enabled":true},{"patternId":"PyLint_E0301","enabled":true},{"patternId":"PyLint_W1306","enabled":true},{"patternId":"PyLint_W1302","enabled":true},{"patternId":"PyLint_E0110","enabled":true},{"patternId":"PyLint_E1125","enabled":true}]} \ No newline at end of file diff --git a/gcloud-java-bigquery/README.md b/gcloud-java-bigquery/README.md index 3f3678f41a04..3387cd8c4f41 100644 --- a/gcloud-java-bigquery/README.md +++ b/gcloud-java-bigquery/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud BigQuery] (https://cloud.google.com/bigq [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-bigquery.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html) @@ -20,22 +22,22 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-bigquery - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-bigquery:0.1.3' +compile 'com.google.gcloud:gcloud-java-bigquery:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-bigquery" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java-bigquery" % "0.1.5" ``` Example Application ------------------- -- [`BigQueryExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality. -Read more about using this application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/BigQueryExample.html). +- [`BigQueryExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java) - A simple command line interface providing some of Cloud BigQuery's functionality. +Read more about using this application on the [`BigQueryExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/bigquery/BigQueryExample.html). Authentication -------------- @@ -114,6 +116,7 @@ with only one string field. Add the following imports at the top of your file: import com.google.gcloud.bigquery.Field; import com.google.gcloud.bigquery.Schema; import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; ``` @@ -127,7 +130,7 @@ Field stringField = Field.of("StringField", Field.Type.string()); Schema schema = Schema.of(stringField); // Create a table StandardTableDefinition tableDefinition = StandardTableDefinition.of(schema); -TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, tableDefinition)); +Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); ``` #### Loading data into a table @@ -199,90 +202,13 @@ while (rowIterator.hasNext()) { ``` #### Complete source code -Here we put together all the code shown above into one program. This program assumes that you are -running on Compute Engine or from your own desktop. To run this example on App Engine, simply move +In +[InsertDataAndQueryTable.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java) +we put together all the code shown above into one program. The program assumes that you are +running on Compute Engine or from your own desktop. To run the example on App Engine, simply move the code from the main method to your application's servlet class and change the print statements to display on your webpage. -```java -import com.google.gcloud.bigquery.BigQuery; -import com.google.gcloud.bigquery.BigQueryOptions; -import com.google.gcloud.bigquery.DatasetInfo; -import com.google.gcloud.bigquery.Field; -import com.google.gcloud.bigquery.FieldValue; -import com.google.gcloud.bigquery.InsertAllRequest; -import com.google.gcloud.bigquery.InsertAllResponse; -import com.google.gcloud.bigquery.QueryRequest; -import com.google.gcloud.bigquery.QueryResponse; -import com.google.gcloud.bigquery.Schema; -import com.google.gcloud.bigquery.StandardTableDefinition; -import com.google.gcloud.bigquery.TableId; -import com.google.gcloud.bigquery.TableInfo; - -import java.util.HashMap; -import java.util.Iterator; -import java.util.List; -import java.util.Map; - -public class GcloudBigQueryExample { - - public static void main(String[] args) throws InterruptedException { - - // Create a service instance - BigQuery bigquery = BigQueryOptions.defaultInstance().service(); - - // Create a dataset - String datasetId = "my_dataset_id"; - bigquery.create(DatasetInfo.builder(datasetId).build()); - - TableId tableId = TableId.of(datasetId, "my_table_id"); - // Table field definition - Field stringField = Field.of("StringField", Field.Type.string()); - // Table schema definition - Schema schema = Schema.of(stringField); - // Create a table - StandardTableDefinition tableDefinition = StandardTableDefinition.of(schema); - TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, tableDefinition)); - - // Define rows to insert - Map firstRow = new HashMap<>(); - Map secondRow = new HashMap<>(); - firstRow.put("StringField", "value1"); - secondRow.put("StringField", "value2"); - // Create an insert request - InsertAllRequest insertRequest = InsertAllRequest.builder(tableId) - .addRow(firstRow) - .addRow(secondRow) - .build(); - // Insert rows - InsertAllResponse insertResponse = bigquery.insertAll(insertRequest); - // Check if errors occurred - if (insertResponse.hasErrors()) { - System.out.println("Errors occurred while inserting rows"); - } - - // Create a query request - QueryRequest queryRequest = - QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id") - .maxWaitTime(60000L) - .maxResults(1000L) - .build(); - // Request query to be executed and wait for results - QueryResponse queryResponse = bigquery.query(queryRequest); - while (!queryResponse.jobCompleted()) { - Thread.sleep(1000L); - queryResponse = bigquery.getQueryResults(queryResponse.jobId()); - } - // Read rows - Iterator> rowIterator = queryResponse.result().iterateAll(); - System.out.println("Table rows:"); - while (rowIterator.hasNext()) { - System.out.println(rowIterator.next()); - } - } -} -``` - Troubleshooting --------------- diff --git a/gcloud-java-bigquery/pom.xml b/gcloud-java-bigquery/pom.xml index a5d711abf610..9a2137cb987d 100644 --- a/gcloud-java-bigquery/pom.xml +++ b/gcloud-java-bigquery/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-bigquery jar GCloud Java bigquery @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT gcloud-java-bigquery @@ -31,13 +30,13 @@ com.google.apis google-api-services-bigquery - v2-rev254-1.21.0 + v2-rev270-1.21.0 compile - - com.google.guava - guava-jdk5 - + + com.google.guava + guava-jdk5 + @@ -49,7 +48,7 @@ org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java index b8e9926ce8c8..b8e1a817c836 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Acl.java @@ -325,6 +325,8 @@ Access toPb() { */ public static final class View extends Entity { + private static final long serialVersionUID = -6851072781269419383L; + private final TableId id; /** diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java index a1b23aba4d5d..986c595e350d 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQuery.java @@ -206,7 +206,7 @@ private DatasetOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the dataset's fields to be returned by the RPC call. If this * option is not provided all dataset's fields are returned. {@code DatasetOption.fields} can - * be used to specify only the fields of interest. {@link DatasetInfo#datasetId()} is always + * be used to specify only the fields of interest. {@link Dataset#datasetId()} is always * returned, even if not specified. */ public static DatasetOption fields(DatasetField... fields) { @@ -275,8 +275,8 @@ private TableOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the table's fields to be returned by the RPC call. If this * option is not provided all table's fields are returned. {@code TableOption.fields} can be - * used to specify only the fields of interest. {@link TableInfo#tableId()} and type (which is - * part of {@link TableInfo#definition()}) are always returned, even if not specified. + * used to specify only the fields of interest. {@link Table#tableId()} and type (which is part + * of {@link Table#definition()}) are always returned, even if not specified. */ public static TableOption fields(TableField... fields) { return new TableOption(BigQueryRpc.Option.FIELDS, TableField.selector(fields)); @@ -369,7 +369,7 @@ public static JobListOption startPageToken(String pageToken) { /** * Returns an option to specify the job's fields to be returned by the RPC call. If this option * is not provided all job's fields are returned. {@code JobOption.fields()} can be used to - * specify only the fields of interest. {@link JobInfo#jobId()}, {@link JobStatus#state()}, + * specify only the fields of interest. {@link Job#jobId()}, {@link JobStatus#state()}, * {@link JobStatus#error()} as well as type-specific configuration (e.g. * {@link QueryJobConfiguration#query()} for Query Jobs) are always returned, even if not * specified. {@link JobField#SELF_LINK} and {@link JobField#ETAG} can not be selected when @@ -397,7 +397,7 @@ private JobOption(BigQueryRpc.Option option, Object value) { /** * Returns an option to specify the job's fields to be returned by the RPC call. If this option * is not provided all job's fields are returned. {@code JobOption.fields()} can be used to - * specify only the fields of interest. {@link JobInfo#jobId()} as well as type-specific + * specify only the fields of interest. {@link Job#jobId()} as well as type-specific * configuration (e.g. {@link QueryJobConfiguration#query()} for Query Jobs) are always * returned, even if not specified. */ @@ -457,46 +457,45 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * * @throws BigQueryException upon failure */ - DatasetInfo create(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; + Dataset create(DatasetInfo dataset, DatasetOption... options); /** * Creates a new table. * * @throws BigQueryException upon failure */ - TableInfo create(TableInfo table, TableOption... options) throws BigQueryException; + Table create(TableInfo table, TableOption... options); /** * Creates a new job. * * @throws BigQueryException upon failure */ - JobInfo create(JobInfo job, JobOption... options) throws BigQueryException; + Job create(JobInfo job, JobOption... options); /** * Returns the requested dataset or {@code null} if not found. * * @throws BigQueryException upon failure */ - DatasetInfo getDataset(String datasetId, DatasetOption... options) throws BigQueryException; + Dataset getDataset(String datasetId, DatasetOption... options); /** * Returns the requested dataset or {@code null} if not found. * * @throws BigQueryException upon failure */ - DatasetInfo getDataset(DatasetId datasetId, DatasetOption... options) throws BigQueryException; + Dataset getDataset(DatasetId datasetId, DatasetOption... options); /** * Lists the project's datasets. This method returns partial information on each dataset - * ({@link DatasetInfo#datasetId()}, {@link DatasetInfo#friendlyName()} and - * {@link DatasetInfo#id()}). To get complete information use either - * {@link #getDataset(String, DatasetOption...)} or + * ({@link Dataset#datasetId()}, {@link Dataset#friendlyName()} and {@link Dataset#id()}). To get + * complete information use either {@link #getDataset(String, DatasetOption...)} or * {@link #getDataset(DatasetId, DatasetOption...)}. * * @throws BigQueryException upon failure */ - Page listDatasets(DatasetListOption... options) throws BigQueryException; + Page listDatasets(DatasetListOption... options); /** * Deletes the requested dataset. @@ -504,7 +503,7 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * @return {@code true} if dataset was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean delete(String datasetId, DatasetDeleteOption... options) throws BigQueryException; + boolean delete(String datasetId, DatasetDeleteOption... options); /** * Deletes the requested dataset. @@ -512,7 +511,7 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * @return {@code true} if dataset was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean delete(DatasetId datasetId, DatasetDeleteOption... options) throws BigQueryException; + boolean delete(DatasetId datasetId, DatasetDeleteOption... options); /** * Deletes the requested table. @@ -520,7 +519,7 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * @return {@code true} if table was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean delete(String datasetId, String tableId) throws BigQueryException; + boolean delete(String datasetId, String tableId); /** * Deletes the requested table. @@ -528,68 +527,64 @@ public static QueryResultsOption maxWaitTime(long maxWaitTime) { * @return {@code true} if table was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean delete(TableId tableId) throws BigQueryException; + boolean delete(TableId tableId); /** * Updates dataset information. * * @throws BigQueryException upon failure */ - DatasetInfo update(DatasetInfo dataset, DatasetOption... options) throws BigQueryException; + Dataset update(DatasetInfo dataset, DatasetOption... options); /** * Updates table information. * * @throws BigQueryException upon failure */ - TableInfo update(TableInfo table, TableOption... options) throws BigQueryException; + Table update(TableInfo table, TableOption... options); /** * Returns the requested table or {@code null} if not found. * * @throws BigQueryException upon failure */ - TableInfo getTable(String datasetId, String tableId, TableOption... options) - throws BigQueryException; + Table getTable(String datasetId, String tableId, TableOption... options); /** * Returns the requested table or {@code null} if not found. * * @throws BigQueryException upon failure */ - TableInfo getTable(TableId tableId, TableOption... options) - throws BigQueryException; + Table getTable(TableId tableId, TableOption... options); /** * Lists the tables in the dataset. This method returns partial information on each table - * ({@link TableInfo#tableId()}, {@link TableInfo#friendlyName()}, {@link TableInfo#id()} and - * type, which is part of {@link TableInfo#definition()}). To get complete information use either + * ({@link Table#tableId()}, {@link Table#friendlyName()}, {@link Table#id()} and type, which + * is part of {@link Table#definition()}). To get complete information use either * {@link #getTable(TableId, TableOption...)} or * {@link #getTable(String, String, TableOption...)}. * * @throws BigQueryException upon failure */ - Page listTables(String datasetId, TableListOption... options) - throws BigQueryException; + Page listTables(String datasetId, TableListOption... options); /** * Lists the tables in the dataset. This method returns partial information on each table - * ({@link TableInfo#tableId()}, {@link TableInfo#friendlyName()}, {@link TableInfo#id()} and - * type, which is part of {@link TableInfo#definition()}). To get complete information use either + * ({@link Table#tableId()}, {@link Table#friendlyName()}, {@link Table#id()} and type, which + * is part of {@link Table#definition()}). To get complete information use either * {@link #getTable(TableId, TableOption...)} or * {@link #getTable(String, String, TableOption...)}. * * @throws BigQueryException upon failure */ - Page listTables(DatasetId datasetId, TableListOption... options) - throws BigQueryException; + Page
listTables(DatasetId datasetId, TableListOption... options); /** * Sends an insert all request. * * @throws BigQueryException upon failure */ - InsertAllResponse insertAll(InsertAllRequest request) throws BigQueryException; + InsertAllResponse insertAll(InsertAllRequest request); /** * Lists the table's rows. @@ -597,36 +592,35 @@ Page listTables(DatasetId datasetId, TableListOption... options) * @throws BigQueryException upon failure */ Page> listTableData(String datasetId, String tableId, - TableDataListOption... options) throws BigQueryException; + TableDataListOption... options); /** * Lists the table's rows. * * @throws BigQueryException upon failure */ - Page> listTableData(TableId tableId, TableDataListOption... options) - throws BigQueryException; + Page> listTableData(TableId tableId, TableDataListOption... options); /** * Returns the requested job or {@code null} if not found. * * @throws BigQueryException upon failure */ - JobInfo getJob(String jobId, JobOption... options) throws BigQueryException; + Job getJob(String jobId, JobOption... options); /** * Returns the requested job or {@code null} if not found. * * @throws BigQueryException upon failure */ - JobInfo getJob(JobId jobId, JobOption... options) throws BigQueryException; + Job getJob(JobId jobId, JobOption... options); /** * Lists the jobs. * * @throws BigQueryException upon failure */ - Page listJobs(JobListOption... options) throws BigQueryException; + Page listJobs(JobListOption... options); /** * Sends a job cancel request. This call will return immediately. The job status can then be @@ -637,7 +631,7 @@ Page> listTableData(TableId tableId, TableDataListOption... opt * found * @throws BigQueryException upon failure */ - boolean cancel(String jobId) throws BigQueryException; + boolean cancel(String jobId); /** * Sends a job cancel request. This call will return immediately. The job status can then be @@ -648,21 +642,21 @@ Page> listTableData(TableId tableId, TableDataListOption... opt * found * @throws BigQueryException upon failure */ - boolean cancel(JobId tableId) throws BigQueryException; + boolean cancel(JobId tableId); /** * Runs the query associated with the request. * * @throws BigQueryException upon failure */ - QueryResponse query(QueryRequest request) throws BigQueryException; + QueryResponse query(QueryRequest request); /** * Returns results of the query associated with the provided job. * * @throws BigQueryException upon failure */ - QueryResponse getQueryResults(JobId job, QueryResultsOption... options) throws BigQueryException; + QueryResponse getQueryResults(JobId job, QueryResultsOption... options); /** * Returns a channel to write data to be inserted into a BigQuery table. Data format and other diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java index de74bdcac89c..ce881c6ea079 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/BigQueryImpl.java @@ -19,10 +19,7 @@ import static com.google.common.base.Preconditions.checkArgument; import static com.google.gcloud.RetryHelper.runWithRetries; -import com.google.api.services.bigquery.model.Dataset; import com.google.api.services.bigquery.model.GetQueryResultsResponse; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.Table; import com.google.api.services.bigquery.model.TableDataInsertAllRequest; import com.google.api.services.bigquery.model.TableDataInsertAllRequest.Rows; import com.google.api.services.bigquery.model.TableRow; @@ -46,7 +43,7 @@ final class BigQueryImpl extends BaseService implements BigQuery { - private static class DatasetPageFetcher implements NextPageFetcher { + private static class DatasetPageFetcher implements NextPageFetcher { private static final long serialVersionUID = -3057564042439021278L; private final Map requestOptions; @@ -60,12 +57,12 @@ private static class DatasetPageFetcher implements NextPageFetcher } @Override - public Page nextPage() { + public Page nextPage() { return listDatasets(serviceOptions, requestOptions); } } - private static class TablePageFetcher implements NextPageFetcher { + private static class TablePageFetcher implements NextPageFetcher
{ private static final long serialVersionUID = 8611248840504201187L; private final Map requestOptions; @@ -81,12 +78,12 @@ private static class TablePageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page
nextPage() { return listTables(dataset, serviceOptions, requestOptions); } } - private static class JobPageFetcher implements NextPageFetcher { + private static class JobPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 8536533282558245472L; private final Map requestOptions; @@ -100,7 +97,7 @@ private static class JobPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listJobs(serviceOptions, requestOptions); } } @@ -156,109 +153,119 @@ public QueryResult nextPage() { } @Override - public DatasetInfo create(DatasetInfo dataset, DatasetOption... options) - throws BigQueryException { - final Dataset datasetPb = dataset.setProjectId(options().projectId()).toPb(); + public Dataset create(DatasetInfo dataset, DatasetOption... options) { + final com.google.api.services.bigquery.model.Dataset datasetPb = + dataset.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return DatasetInfo.fromPb(runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.create(datasetPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Dataset.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.create(datasetPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public TableInfo create(TableInfo table, TableOption... options) - throws BigQueryException { - final Table tablePb = table.setProjectId(options().projectId()).toPb(); + public Table create(TableInfo table, TableOption... options) { + final com.google.api.services.bigquery.model.Table tablePb = + table.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return TableInfo.fromPb(runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.create(tablePb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Table.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.create(tablePb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public JobInfo create(JobInfo job, JobOption... options) throws BigQueryException { - final Job jobPb = job.setProjectId(options().projectId()).toPb(); + public Job create(JobInfo job, JobOption... options) { + final com.google.api.services.bigquery.model.Job jobPb = + job.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return JobInfo.fromPb(runWithRetries(new Callable() { - @Override - public Job call() { - return bigQueryRpc.create(jobPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Job.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Job call() { + return bigQueryRpc.create(jobPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public DatasetInfo getDataset(String datasetId, DatasetOption... options) - throws BigQueryException { + public Dataset getDataset(String datasetId, DatasetOption... options) { return getDataset(DatasetId.of(datasetId), options); } @Override - public DatasetInfo getDataset(final DatasetId datasetId, DatasetOption... options) - throws BigQueryException { + public Dataset getDataset(final DatasetId datasetId, DatasetOption... options) { final Map optionsMap = optionMap(options); try { - Dataset answer = runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.getDataset(datasetId.dataset(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : DatasetInfo.fromPb(answer); + com.google.api.services.bigquery.model.Dataset answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.getDataset(datasetId.dataset(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Dataset.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listDatasets(DatasetListOption... options) throws BigQueryException { + public Page listDatasets(DatasetListOption... options) { return listDatasets(options(), optionMap(options)); } - private static Page listDatasets(final BigQueryOptions serviceOptions, + private static Page listDatasets(final BigQueryOptions serviceOptions, final Map optionsMap) { try { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { - @Override - public BigQueryRpc.Tuple> call() { - return serviceOptions.rpc().listDatasets(optionsMap); - } - }, serviceOptions.retryParams(), EXCEPTION_HANDLER); + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { + @Override + public BigQueryRpc.Tuple> call() { + return serviceOptions.rpc().listDatasets(optionsMap); + } + }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); return new PageImpl<>(new DatasetPageFetcher(serviceOptions, cursor, optionsMap), cursor, - Iterables.transform(result.y(), DatasetInfo.FROM_PB_FUNCTION)); + Iterables.transform(result.y(), + new Function() { + @Override + public Dataset apply(com.google.api.services.bigquery.model.Dataset dataset) { + return Dataset.fromPb(serviceOptions.service(), dataset); + } + })); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public boolean delete(String datasetId, DatasetDeleteOption... options) throws BigQueryException { + public boolean delete(String datasetId, DatasetDeleteOption... options) { return delete(DatasetId.of(datasetId), options); } @Override - public boolean delete(final DatasetId datasetId, DatasetDeleteOption... options) - throws BigQueryException { + public boolean delete(final DatasetId datasetId, DatasetDeleteOption... options) { final Map optionsMap = optionMap(options); try { return runWithRetries(new Callable() { @@ -273,12 +280,12 @@ public Boolean call() { } @Override - public boolean delete(String datasetId, String tableId) throws BigQueryException { + public boolean delete(String datasetId, String tableId) { return delete(TableId.of(datasetId, tableId)); } @Override - public boolean delete(final TableId tableId) throws BigQueryException { + public boolean delete(final TableId tableId) { try { return runWithRetries(new Callable() { @Override @@ -292,87 +299,93 @@ public Boolean call() { } @Override - public DatasetInfo update(DatasetInfo dataset, DatasetOption... options) - throws BigQueryException { - final Dataset datasetPb = dataset.setProjectId(options().projectId()).toPb(); + public Dataset update(DatasetInfo dataset, DatasetOption... options) { + final com.google.api.services.bigquery.model.Dataset datasetPb = + dataset.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return DatasetInfo.fromPb(runWithRetries(new Callable() { - @Override - public Dataset call() { - return bigQueryRpc.patch(datasetPb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Dataset.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Dataset call() { + return bigQueryRpc.patch(datasetPb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public TableInfo update(TableInfo table, TableOption... options) - throws BigQueryException { - final Table tablePb = table.setProjectId(options().projectId()).toPb(); + public Table update(TableInfo table, TableOption... options) { + final com.google.api.services.bigquery.model.Table tablePb = + table.setProjectId(options().projectId()).toPb(); final Map optionsMap = optionMap(options); try { - return TableInfo.fromPb(runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.patch(tablePb, optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER)); + return Table.fromPb(this, + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.patch(tablePb, optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER)); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public TableInfo getTable(final String datasetId, final String tableId, - TableOption... options) throws BigQueryException { + public Table getTable(final String datasetId, final String tableId, TableOption... options) { return getTable(TableId.of(datasetId, tableId), options); } @Override - public TableInfo getTable(final TableId tableId, TableOption... options) - throws BigQueryException { + public Table getTable(final TableId tableId, TableOption... options) { final Map optionsMap = optionMap(options); try { - Table answer = runWithRetries(new Callable
() { - @Override - public Table call() { - return bigQueryRpc.getTable(tableId.dataset(), tableId.table(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : TableInfo.fromPb(answer); + com.google.api.services.bigquery.model.Table answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Table call() { + return bigQueryRpc.getTable(tableId.dataset(), tableId.table(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Table.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listTables(String datasetId, TableListOption... options) - throws BigQueryException { + public Page
listTables(String datasetId, TableListOption... options) { return listTables(datasetId, options(), optionMap(options)); } @Override - public Page listTables(DatasetId datasetId, TableListOption... options) - throws BigQueryException { + public Page
listTables(DatasetId datasetId, TableListOption... options) { return listTables(datasetId.dataset(), options(), optionMap(options)); } - private static Page listTables(final String datasetId, final BigQueryOptions + private static Page
listTables(final String datasetId, final BigQueryOptions serviceOptions, final Map optionsMap) { try { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { @Override - public BigQueryRpc.Tuple> call() { - return serviceOptions.rpc().listTables(datasetId, optionsMap); - } + public BigQueryRpc.Tuple> + call() { + return serviceOptions.rpc().listTables(datasetId, optionsMap); + } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable tables = Iterables.transform(result.y(), - TableInfo.FROM_PB_FUNCTION); + Iterable
tables = Iterables.transform(result.y(), + new Function() { + @Override + public Table apply(com.google.api.services.bigquery.model.Table table) { + return Table.fromPb(serviceOptions.service(), table); + } + }); return new PageImpl<>(new TablePageFetcher(datasetId, serviceOptions, cursor, optionsMap), cursor, tables); } catch (RetryHelper.RetryHelperException e) { @@ -381,7 +394,7 @@ public BigQueryRpc.Tuple> call() { } @Override - public InsertAllResponse insertAll(InsertAllRequest request) throws BigQueryException { + public InsertAllResponse insertAll(InsertAllRequest request) { final TableId tableId = request.table(); final TableDataInsertAllRequest requestPb = new TableDataInsertAllRequest(); requestPb.setIgnoreUnknownValues(request.ignoreUnknownValues()); @@ -400,13 +413,12 @@ public Rows apply(RowToInsert rowToInsert) { @Override public Page> listTableData(String datasetId, String tableId, - TableDataListOption... options) throws BigQueryException { + TableDataListOption... options) { return listTableData(TableId.of(datasetId, tableId), options(), optionMap(options)); } @Override - public Page> listTableData(TableId tableId, TableDataListOption... options) - throws BigQueryException { + public Page> listTableData(TableId tableId, TableDataListOption... options) { return listTableData(tableId, options(), optionMap(options)); } @@ -441,53 +453,61 @@ public List apply(TableRow rowPb) { } @Override - public JobInfo getJob(String jobId, JobOption... options) throws BigQueryException { + public Job getJob(String jobId, JobOption... options) { return getJob(JobId.of(jobId), options); } @Override - public JobInfo getJob(final JobId jobId, JobOption... options) - throws BigQueryException { + public Job getJob(final JobId jobId, JobOption... options) { final Map optionsMap = optionMap(options); try { - Job answer = runWithRetries(new Callable() { - @Override - public Job call() { - return bigQueryRpc.getJob(jobId.job(), optionsMap); - } - }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : JobInfo.fromPb(answer); + com.google.api.services.bigquery.model.Job answer = + runWithRetries(new Callable() { + @Override + public com.google.api.services.bigquery.model.Job call() { + return bigQueryRpc.getJob(jobId.job(), optionsMap); + } + }, options().retryParams(), EXCEPTION_HANDLER); + return answer == null ? null : Job.fromPb(this, answer); } catch (RetryHelper.RetryHelperException e) { throw BigQueryException.translateAndThrow(e); } } @Override - public Page listJobs(JobListOption... options) throws BigQueryException { + public Page listJobs(JobListOption... options) { return listJobs(options(), optionMap(options)); } - private static Page listJobs(final BigQueryOptions serviceOptions, + private static Page listJobs(final BigQueryOptions serviceOptions, final Map optionsMap) { - BigQueryRpc.Tuple> result = - runWithRetries(new Callable>>() { + BigQueryRpc.Tuple> result = + runWithRetries(new Callable>>() { @Override - public BigQueryRpc.Tuple> call() { + public BigQueryRpc.Tuple> + call() { return serviceOptions.rpc().listJobs(optionsMap); } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable jobs = Iterables.transform(result.y(), JobInfo.FROM_PB_FUNCTION); + Iterable jobs = Iterables.transform(result.y(), + new Function() { + @Override + public Job apply(com.google.api.services.bigquery.model.Job job) { + return Job.fromPb(serviceOptions.service(), job); + } + }); return new PageImpl<>(new JobPageFetcher(serviceOptions, cursor, optionsMap), cursor, jobs); } @Override - public boolean cancel(String jobId) throws BigQueryException { + public boolean cancel(String jobId) { return cancel(JobId.of(jobId)); } @Override - public boolean cancel(final JobId jobId) throws BigQueryException { + public boolean cancel(final JobId jobId) { try { return runWithRetries(new Callable() { @Override @@ -501,7 +521,7 @@ public Boolean call() { } @Override - public QueryResponse query(final QueryRequest request) throws BigQueryException { + public QueryResponse query(final QueryRequest request) { try { com.google.api.services.bigquery.model.QueryResponse results = runWithRetries(new Callable() { @@ -540,8 +560,7 @@ public com.google.api.services.bigquery.model.QueryResponse call() { } @Override - public QueryResponse getQueryResults(JobId job, QueryResultsOption... options) - throws BigQueryException { + public QueryResponse getQueryResults(JobId job, QueryResultsOption... options) { Map optionsMap = optionMap(options); return getQueryResults(job, options(), optionsMap); } @@ -595,6 +614,7 @@ private static QueryResult.Builder transformQueryResults(JobId jobId, List
Objects of this class are immutable. Operations that modify the dataset like {@link #update} * return a new object. To get a {@code Dataset} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Dataset} adds a layer of service-related functionality over + * {@link DatasetInfo}. *

*/ -public final class Dataset { +public final class Dataset extends DatasetInfo { - private final BigQuery bigquery; - private final DatasetInfo info; + private static final long serialVersionUID = -4272921483363065593L; - private static class TablePageFetcher implements PageImpl.NextPageFetcher
{ + private final BigQueryOptions options; + private transient BigQuery bigquery; - private static final long serialVersionUID = 6906197848579250598L; + /** + * A builder for {@code Dataset} objects. + */ + public static final class Builder extends DatasetInfo.Builder { + + private final BigQuery bigquery; + private final DatasetInfo.BuilderImpl infoBuilder; - private final BigQueryOptions options; - private final Page infoPage; + Builder(BigQuery bigquery, DatasetId datasetId) { + this.bigquery = bigquery; + this.infoBuilder = new DatasetInfo.BuilderImpl(); + this.infoBuilder.datasetId(datasetId); + } - TablePageFetcher(BigQueryOptions options, Page infoPage) { - this.options = options; - this.infoPage = infoPage; + Builder(Dataset dataset) { + this.bigquery = dataset.bigquery; + this.infoBuilder = new DatasetInfo.BuilderImpl(dataset); } @Override - public Page
nextPage() { - Page nextInfoPage = infoPage.nextPage(); - return new PageImpl<>(new TablePageFetcher(options, nextInfoPage), - nextInfoPage.nextPageCursor(), new LazyTableIterable(options, nextInfoPage.values())); + public Builder datasetId(DatasetId datasetId) { + infoBuilder.datasetId(datasetId); + return this; } - } - private static class LazyTableIterable implements Iterable
, Serializable { + @Override + public Builder acl(List acl) { + infoBuilder.acl(acl); + return this; + } - private static final long serialVersionUID = 3312744215731674032L; + @Override + Builder creationTime(Long creationTime) { + infoBuilder.creationTime(creationTime); + return this; + } - private final BigQueryOptions options; - private final Iterable infoIterable; - private transient BigQuery bigquery; + @Override + public Builder defaultTableLifetime(Long defaultTableLifetime) { + infoBuilder.defaultTableLifetime(defaultTableLifetime); + return this; + } - public LazyTableIterable(BigQueryOptions options, Iterable infoIterable) { - this.options = options; - this.infoIterable = infoIterable; - this.bigquery = options.service(); + @Override + public Builder description(String description) { + infoBuilder.description(description); + return this; } - private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { - in.defaultReadObject(); - this.bigquery = options.service(); + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; } @Override - public Iterator
iterator() { - return Iterators.transform(infoIterable.iterator(), new Function() { - @Override - public Table apply(TableInfo tableInfo) { - return new Table(bigquery, tableInfo); - } - }); + public Builder friendlyName(String friendlyName) { + infoBuilder.friendlyName(friendlyName); + return this; } @Override - public int hashCode() { - return Objects.hash(options, infoIterable); + Builder id(String id) { + infoBuilder.id(id); + return this; } @Override - public boolean equals(Object obj) { - if (!(obj instanceof LazyTableIterable)) { - return false; - } - LazyTableIterable other = (LazyTableIterable) obj; - return Objects.equals(options, other.options) - && Objects.equals(infoIterable, other.infoIterable); + Builder lastModified(Long lastModified) { + infoBuilder.lastModified(lastModified); + return this; } - } - /** - * Constructs a {@code Dataset} object for the provided {@code DatasetInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info dataset's info - */ - public Dataset(BigQuery bigquery, DatasetInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + @Override + public Builder location(String location) { + infoBuilder.location(location); + return this; + } - /** - * Creates a {@code Dataset} object for the provided dataset's user-defined id. Performs an RPC - * call to get the latest dataset information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param dataset dataset's user-defined id - * @param options dataset options - * @return the {@code Dataset} object or {@code null} if not found - * @throws BigQueryException upon failure - */ - public static Dataset get(BigQuery bigquery, String dataset, BigQuery.DatasetOption... options) { - DatasetInfo info = bigquery.getDataset(dataset, options); - return info != null ? new Dataset(bigquery, info) : null; + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Dataset build() { + return new Dataset(bigquery, infoBuilder); + } } - /** - * Returns the dataset's information. - */ - public DatasetInfo info() { - return info; + Dataset(BigQuery bigquery, DatasetInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -149,7 +145,7 @@ public DatasetInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getDataset(info.datasetId(), BigQuery.DatasetOption.fields()) != null; + return bigquery.getDataset(datasetId(), BigQuery.DatasetOption.fields()) != null; } /** @@ -161,23 +157,19 @@ public boolean exists() { * @throws BigQueryException upon failure */ public Dataset reload(BigQuery.DatasetOption... options) { - return Dataset.get(bigquery, info.datasetId().dataset(), options); + return bigquery.getDataset(datasetId().dataset(), options); } /** - * Updates the dataset's information. Dataset's user-defined id cannot be changed. A new - * {@code Dataset} object is returned. + * Updates the dataset's information with this dataset's information. Dataset's user-defined id + * cannot be changed. A new {@code Dataset} object is returned. * - * @param datasetInfo new dataset's information. User-defined id must match the one of the current - * dataset * @param options dataset options * @return a {@code Dataset} object with updated information * @throws BigQueryException upon failure */ - public Dataset update(DatasetInfo datasetInfo, BigQuery.DatasetOption... options) { - checkArgument(Objects.equals(datasetInfo.datasetId().dataset(), - info.datasetId().dataset()), "Dataset's user-defined ids must match"); - return new Dataset(bigquery, bigquery.update(datasetInfo, options)); + public Dataset update(BigQuery.DatasetOption... options) { + return bigquery.update(this, options); } /** @@ -187,7 +179,7 @@ public Dataset update(DatasetInfo datasetInfo, BigQuery.DatasetOption... options * @throws BigQueryException upon failure */ public boolean delete() { - return bigquery.delete(info.datasetId()); + return bigquery.delete(datasetId()); } /** @@ -197,10 +189,7 @@ public boolean delete() { * @throws BigQueryException upon failure */ public Page
list(BigQuery.TableListOption... options) { - Page infoPage = bigquery.listTables(info.datasetId(), options); - BigQueryOptions bigqueryOptions = bigquery.options(); - return new PageImpl<>(new TablePageFetcher(bigqueryOptions, infoPage), - infoPage.nextPageCursor(), new LazyTableIterable(bigqueryOptions, infoPage.values())); + return bigquery.listTables(datasetId(), options); } /** @@ -211,8 +200,7 @@ public Page
list(BigQuery.TableListOption... options) { * @throws BigQueryException upon failure */ public Table get(String table, BigQuery.TableOption... options) { - TableInfo tableInfo = bigquery.getTable(TableId.of(info.datasetId().dataset(), table), options); - return tableInfo != null ? new Table(bigquery, tableInfo) : null; + return bigquery.getTable(TableId.of(datasetId().dataset(), table), options); } /** @@ -225,8 +213,8 @@ public Table get(String table, BigQuery.TableOption... options) { * @throws BigQueryException upon failure */ public Table create(String table, TableDefinition definition, BigQuery.TableOption... options) { - TableInfo tableInfo = TableInfo.of(TableId.of(info.datasetId().dataset(), table), definition); - return new Table(bigquery, bigquery.create(tableInfo, options)); + TableInfo tableInfo = TableInfo.of(TableId.of(datasetId().dataset(), table), definition); + return bigquery.create(tableInfo, options); } /** @@ -235,4 +223,31 @@ public Table create(String table, TableDefinition definition, BigQuery.TableOpti public BigQuery bigquery() { return bigquery; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Dataset + && Objects.equals(toPb(), ((Dataset) obj).toPb()) + && Objects.equals(options, ((Dataset) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Dataset fromPb(BigQuery bigquery, + com.google.api.services.bigquery.model.Dataset datasetPb) { + return new Dataset(bigquery, new DatasetInfo.BuilderImpl(datasetPb)); + } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java index c6330308c8ce..aa767b97631b 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/DatasetInfo.java @@ -39,7 +39,7 @@ * @see * Managing Jobs, Datasets, and Projects */ -public final class DatasetInfo implements Serializable { +public class DatasetInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -70,7 +70,72 @@ public Dataset apply(DatasetInfo datasetInfo) { private final String location; private final String selfLink; - public static final class Builder { + /** + * A builder for {@code DatasetInfo} objects. + */ + public abstract static class Builder { + + /** + * Sets the dataset identity. + */ + public abstract Builder datasetId(DatasetId datasetId); + + /** + * Sets the dataset's access control configuration. + * + * @see Access Control + */ + public abstract Builder acl(List acl); + + abstract Builder creationTime(Long creationTime); + + /** + * Sets the default lifetime of all tables in the dataset, in milliseconds. The minimum value is + * 3600000 milliseconds (one hour). Once this property is set, all newly-created tables in the + * dataset will have an expirationTime property set to the creation time plus the value in this + * property, and changing the value will only affect new tables, not existing ones. When the + * expirationTime for a given table is reached, that table will be deleted automatically. If a + * table's expirationTime is modified or removed before the table expires, or if you provide an + * explicit expirationTime when creating a table, that value takes precedence over the default + * expiration time indicated by this property. This property is experimental and might be + * subject to change or removed. + */ + public abstract Builder defaultTableLifetime(Long defaultTableLifetime); + + /** + * Sets a user-friendly description for the dataset. + */ + public abstract Builder description(String description); + + abstract Builder etag(String etag); + + /** + * Sets a user-friendly name for the dataset. + */ + public abstract Builder friendlyName(String friendlyName); + + abstract Builder id(String id); + + abstract Builder lastModified(Long lastModified); + + /** + * Sets the geographic location where the dataset should reside. This property is experimental + * and might be subject to change or removed. + * + * @see Dataset + * Location + */ + public abstract Builder location(String location); + + abstract Builder selfLink(String selfLink); + + /** + * Creates a {@code DatasetInfo} object. + */ + public abstract DatasetInfo build(); + } + + static final class BuilderImpl extends Builder { private DatasetId datasetId; private List acl; @@ -84,9 +149,9 @@ public static final class Builder { private String location; private String selfLink; - private Builder() {} + BuilderImpl() {} - private Builder(DatasetInfo datasetInfo) { + BuilderImpl(DatasetInfo datasetInfo) { this.datasetId = datasetInfo.datasetId; this.acl = datasetInfo.acl; this.creationTime = datasetInfo.creationTime; @@ -100,103 +165,103 @@ private Builder(DatasetInfo datasetInfo) { this.selfLink = datasetInfo.selfLink; } - /** - * Sets the dataset identity. - */ + BuilderImpl(com.google.api.services.bigquery.model.Dataset datasetPb) { + if (datasetPb.getDatasetReference() != null) { + this.datasetId = DatasetId.fromPb(datasetPb.getDatasetReference()); + } + if (datasetPb.getAccess() != null) { + this.acl = Lists.transform(datasetPb.getAccess(), new Function() { + @Override + public Acl apply(Dataset.Access accessPb) { + return Acl.fromPb(accessPb); + } + }); + } + this.creationTime = datasetPb.getCreationTime(); + this.defaultTableLifetime = datasetPb.getDefaultTableExpirationMs(); + this.description = datasetPb.getDescription(); + this.etag = datasetPb.getEtag(); + this.friendlyName = datasetPb.getFriendlyName(); + this.id = datasetPb.getId(); + this.lastModified = datasetPb.getLastModifiedTime(); + this.location = datasetPb.getLocation(); + this.selfLink = datasetPb.getSelfLink(); + } + + @Override public Builder datasetId(DatasetId datasetId) { this.datasetId = checkNotNull(datasetId); return this; } - /** - * Sets the dataset's access control configuration. - * - * @see Access Control - */ + @Override public Builder acl(List acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } + @Override Builder creationTime(Long creationTime) { this.creationTime = creationTime; return this; } - /** - * Sets the default lifetime of all tables in the dataset, in milliseconds. The minimum value is - * 3600000 milliseconds (one hour). Once this property is set, all newly-created tables in the - * dataset will have an expirationTime property set to the creation time plus the value in this - * property, and changing the value will only affect new tables, not existing ones. When the - * expirationTime for a given table is reached, that table will be deleted automatically. If a - * table's expirationTime is modified or removed before the table expires, or if you provide an - * explicit expirationTime when creating a table, that value takes precedence over the default - * expiration time indicated by this property. This property is experimental and might be - * subject to change or removed. - */ + @Override public Builder defaultTableLifetime(Long defaultTableLifetime) { this.defaultTableLifetime = firstNonNull(defaultTableLifetime, Data.nullOf(Long.class)); return this; } - /** - * Sets a user-friendly description for the dataset. - */ + @Override public Builder description(String description) { this.description = firstNonNull(description, Data.nullOf(String.class)); return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } - /** - * Sets a user-friendly name for the dataset. - */ + @Override public Builder friendlyName(String friendlyName) { this.friendlyName = firstNonNull(friendlyName, Data.nullOf(String.class)); return this; } + @Override Builder id(String id) { this.id = id; return this; } + @Override Builder lastModified(Long lastModified) { this.lastModified = lastModified; return this; } - /** - * Sets the geographic location where the dataset should reside. This property is experimental - * and might be subject to change or removed. - * - * @see Dataset - * Location - */ + @Override public Builder location(String location) { this.location = firstNonNull(location, Data.nullOf(String.class)); return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Creates a {@code DatasetInfo} object. - */ + @Override public DatasetInfo build() { return new DatasetInfo(this); } } - private DatasetInfo(Builder builder) { + DatasetInfo(BuilderImpl builder) { datasetId = checkNotNull(builder.datasetId); acl = builder.acl; creationTime = builder.creationTime; @@ -301,10 +366,10 @@ public String selfLink() { } /** - * Returns a builder for the {@code DatasetInfo} object. + * Returns a builder for the dataset object. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -331,7 +396,9 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof DatasetInfo && Objects.equals(toPb(), ((DatasetInfo) obj).toPb()); + return obj != null + && obj.getClass().equals(DatasetInfo.class) + && Objects.equals(toPb(), ((DatasetInfo) obj).toPb()); } DatasetInfo setProjectId(String projectId) { @@ -380,65 +447,27 @@ public Dataset.Access apply(Acl acl) { } /** - * Returns a builder for the DatasetInfo object given it's user-defined id. + * Returns a builder for a {@code DatasetInfo} object given it's identity. */ - public static Builder builder(String datasetId) { - return new Builder().datasetId(DatasetId.of(datasetId)); + public static Builder builder(DatasetId datasetId) { + return new BuilderImpl().datasetId(datasetId); } /** - * Returns a builder for the DatasetInfo object given it's project and user-defined id. + * Returns a builder for a {@code DatasetInfo} object given it's user-defined id. */ - public static Builder builder(String projectId, String datasetId) { - return new Builder().datasetId(DatasetId.of(projectId, datasetId)); + public static Builder builder(String datasetId) { + return builder(DatasetId.of(datasetId)); } /** - * Returns a builder for the DatasetInfo object given it's identity. + * Returns a builder for the DatasetInfo object given it's user-defined project and dataset ids. */ - public static Builder builder(DatasetId datasetId) { - return new Builder().datasetId(datasetId); + public static Builder builder(String projectId, String datasetId) { + return builder(DatasetId.of(projectId, datasetId)); } static DatasetInfo fromPb(Dataset datasetPb) { - Builder builder = builder(datasetPb.getDatasetReference().getProjectId(), - datasetPb.getDatasetReference().getDatasetId()); - if (datasetPb.getAccess() != null) { - builder.acl(Lists.transform(datasetPb.getAccess(), - new Function() { - @Override - public Acl apply(Dataset.Access accessPb) { - return Acl.fromPb(accessPb); - } - })); - } - if (datasetPb.getCreationTime() != null) { - builder.creationTime(datasetPb.getCreationTime()); - } - if (datasetPb.getDefaultTableExpirationMs() != null) { - builder.defaultTableLifetime(datasetPb.getDefaultTableExpirationMs()); - } - if (datasetPb.getDescription() != null) { - builder.description(datasetPb.getDescription()); - } - if (datasetPb.getEtag() != null) { - builder.etag(datasetPb.getEtag()); - } - if (datasetPb.getFriendlyName() != null) { - builder.friendlyName(datasetPb.getFriendlyName()); - } - if (datasetPb.getId() != null) { - builder.id(datasetPb.getId()); - } - if (datasetPb.getLastModifiedTime() != null) { - builder.lastModified(datasetPb.getLastModifiedTime()); - } - if (datasetPb.getLocation() != null) { - builder.location(datasetPb.getLocation()); - } - if (datasetPb.getSelfLink() != null) { - builder.selfLink(datasetPb.getSelfLink()); - } - return builder.build(); + return new BuilderImpl(datasetPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java index 882b1eb7065f..5f396d948f5a 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExternalTableDefinition.java @@ -28,8 +28,8 @@ import java.util.Objects; /** - * Google BigQuery external table type. BigQuery's external tables are tables whose data reside - * outside of BigQuery but can be queried as normal BigQuery tables. External tables are + * Google BigQuery external table definition. BigQuery's external tables are tables whose data + * reside outside of BigQuery but can be queried as normal BigQuery tables. External tables are * experimental and might be subject to change or removed. * * @see Federated Data Sources diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java index d8e57bd17254..7c5a2698b159 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ExtractJobConfiguration.java @@ -225,6 +225,7 @@ ExtractJobConfiguration setProjectId(String projectId) { return toBuilder().sourceTable(sourceTable().setProjectId(projectId)).build(); } + @Override com.google.api.services.bigquery.model.JobConfiguration toPb() { JobConfigurationExtract extractConfigurationPb = new JobConfigurationExtract(); extractConfigurationPb.setDestinationUris(destinationUris); diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java index 24c4b28b7613..8b27c70db782 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/FieldValue.java @@ -20,9 +20,9 @@ import static com.google.common.base.Preconditions.checkState; import com.google.api.client.util.Data; -import com.google.api.client.util.Lists; import com.google.common.base.Function; import com.google.common.base.MoreObjects; +import com.google.common.collect.Lists; import java.io.Serializable; import java.util.List; diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java index 6f39f20e498d..f0d61583f83f 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/InsertAllRequest.java @@ -52,15 +52,15 @@ public class InsertAllRequest implements Serializable { * id used by BigQuery to detect duplicate insertion requests on a best-effort basis. * *

Example usage of creating a row to insert: - *

    {@code
-   *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-   *   Map recordContent = new HashMap();
-   *   recordContent.put("subfieldName1", "value");
-   *   recordContent.put("subfieldName2", repeatedFieldValue);
-   *   Map rowContent = new HashMap();
-   *   rowContent.put("fieldName1", true);
-   *   rowContent.put("fieldName2", recordContent);
-   *   RowToInsert row = new RowToInsert("rowId", rowContent);
+   * 
 {@code
+   * List repeatedFieldValue = Arrays.asList(1L, 2L);
+   * Map recordContent = new HashMap();
+   * recordContent.put("subfieldName1", "value");
+   * recordContent.put("subfieldName2", repeatedFieldValue);
+   * Map rowContent = new HashMap();
+   * rowContent.put("fieldName1", true);
+   * rowContent.put("fieldName2", recordContent);
+   * RowToInsert row = new RowToInsert("rowId", rowContent);
    * }
* * @see
@@ -177,16 +177,16 @@ public Builder addRow(RowToInsert rowToInsert) { * Adds a row to be inserted with associated id. * *

Example usage of adding a row with associated id: - *

    {@code
-     *   InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
-     *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-     *   Map recordContent = new HashMap();
-     *   recordContent.put("subfieldName1", "value");
-     *   recordContent.put("subfieldName2", repeatedFieldValue);
-     *   Map rowContent = new HashMap();
-     *   rowContent.put("fieldName1", true);
-     *   rowContent.put("fieldName2", recordContent);
-     *   builder.addRow("rowId", rowContent);
+     * 
 {@code
+     * InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
+     * List repeatedFieldValue = Arrays.asList(1L, 2L);
+     * Map recordContent = new HashMap();
+     * recordContent.put("subfieldName1", "value");
+     * recordContent.put("subfieldName2", repeatedFieldValue);
+     * Map rowContent = new HashMap();
+     * rowContent.put("fieldName1", true);
+     * rowContent.put("fieldName2", recordContent);
+     * builder.addRow("rowId", rowContent);
      * }
*/ public Builder addRow(String id, Map content) { @@ -198,16 +198,16 @@ public Builder addRow(String id, Map content) { * Adds a row to be inserted without an associated id. * *

Example usage of adding a row without an associated id: - *

    {@code
-     *   InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
-     *   List repeatedFieldValue = Arrays.asList(1L, 2L);
-     *   Map recordContent = new HashMap();
-     *   recordContent.put("subfieldName1", "value");
-     *   recordContent.put("subfieldName2", repeatedFieldValue);
-     *   Map rowContent = new HashMap();
-     *   rowContent.put("fieldName1", true);
-     *   rowContent.put("fieldName2", recordContent);
-     *   builder.addRow(rowContent);
+     * 
 {@code
+     * InsertAllRequest.Builder builder = InsertAllRequest.builder(tableId);
+     * List repeatedFieldValue = Arrays.asList(1L, 2L);
+     * Map recordContent = new HashMap();
+     * recordContent.put("subfieldName1", "value");
+     * recordContent.put("subfieldName2", repeatedFieldValue);
+     * Map rowContent = new HashMap();
+     * rowContent.put("fieldName1", true);
+     * rowContent.put("fieldName2", recordContent);
+     * builder.addRow(rowContent);
      * }
*/ public Builder addRow(Map content) { diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java index c0d7ddc29c37..1e63344a600d 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Job.java @@ -18,50 +18,102 @@ import static com.google.common.base.Preconditions.checkNotNull; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.util.Objects; + /** * A Google BigQuery Job. * *

Objects of this class are immutable. To get a {@code Job} object with the most recent - * information use {@link #reload}. + * information use {@link #reload}. {@code Job} adds a layer of service-related functionality over + * {@link JobInfo}. *

*/ -public final class Job { +public final class Job extends JobInfo { - private final BigQuery bigquery; - private final JobInfo info; + private static final long serialVersionUID = -4324100991693024704L; - /** - * Constructs a {@code Job} object for the provided {@code JobInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info jobs's info - */ - public Job(BigQuery bigquery, JobInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + private final BigQueryOptions options; + private transient BigQuery bigquery; /** - * Creates a {@code Job} object for the provided job's user-defined id. Performs an RPC call to - * get the latest job information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param job job's id, either user-defined or picked by the BigQuery service - * @param options job options - * @return the {@code Job} object or {@code null} if not found - * @throws BigQueryException upon failure + * A builder for {@code Job} objects. */ - public static Job get(BigQuery bigquery, String job, BigQuery.JobOption... options) { - JobInfo info = bigquery.getJob(job, options); - return info != null ? new Job(bigquery, info) : null; + public static final class Builder extends JobInfo.Builder { + + private final BigQuery bigquery; + private final JobInfo.BuilderImpl infoBuilder; + + Builder(BigQuery bigquery, JobConfiguration configuration) { + this.bigquery = bigquery; + this.infoBuilder = new JobInfo.BuilderImpl(); + this.infoBuilder.configuration(configuration); + } + + Builder(Job job) { + this.bigquery = job.bigquery; + this.infoBuilder = new JobInfo.BuilderImpl(job); + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + public Builder jobId(JobId jobId) { + infoBuilder.jobId(jobId); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + Builder status(JobStatus status) { + infoBuilder.status(status); + return this; + } + + @Override + Builder statistics(JobStatistics statistics) { + infoBuilder.statistics(statistics); + return this; + } + + @Override + Builder userEmail(String userEmail) { + infoBuilder.userEmail(userEmail); + return this; + } + + @Override + public Builder configuration(JobConfiguration configuration) { + infoBuilder.configuration(configuration); + return this; + } + + @Override + public Job build() { + return new Job(bigquery, infoBuilder); + } } - /** - * Returns the job's information. - */ - public JobInfo info() { - return info; + Job(BigQuery bigquery, JobInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -71,7 +123,7 @@ public JobInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getJob(info.jobId(), BigQuery.JobOption.fields()) != null; + return bigquery.getJob(jobId(), BigQuery.JobOption.fields()) != null; } /** @@ -90,8 +142,7 @@ public boolean exists() { * @throws BigQueryException upon failure */ public boolean isDone() { - JobInfo job = bigquery.getJob(info.jobId(), - BigQuery.JobOption.fields(BigQuery.JobField.STATUS)); + Job job = bigquery.getJob(jobId(), BigQuery.JobOption.fields(BigQuery.JobField.STATUS)); return job != null && job.status().state() == JobStatus.State.DONE; } @@ -103,7 +154,7 @@ public boolean isDone() { * @throws BigQueryException upon failure */ public Job reload(BigQuery.JobOption... options) { - return Job.get(bigquery, info.jobId().job(), options); + return bigquery.getJob(jobId().job(), options); } /** @@ -114,7 +165,7 @@ public Job reload(BigQuery.JobOption... options) { * @throws BigQueryException upon failure */ public boolean cancel() { - return bigquery.cancel(info.jobId()); + return bigquery.cancel(jobId()); } /** @@ -123,4 +174,30 @@ public boolean cancel() { public BigQuery bigquery() { return bigquery; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Job + && Objects.equals(toPb(), ((Job) obj).toPb()) + && Objects.equals(options, ((Job) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Job fromPb(BigQuery bigquery, com.google.api.services.bigquery.model.Job jobPb) { + return new Job(bigquery, new JobInfo.BuilderImpl(jobPb)); + } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java index 47135b6d97d0..1adf7fabafc1 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/JobInfo.java @@ -32,7 +32,7 @@ * * @see
Jobs */ -public final class JobInfo implements Serializable { +public class JobInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -41,8 +41,18 @@ public JobInfo apply(Job pb) { return JobInfo.fromPb(pb); } }; + private static final long serialVersionUID = -3272941007234620265L; + private final String etag; + private final String id; + private final JobId jobId; + private final String selfLink; + private final JobStatus status; + private final JobStatistics statistics; + private final String userEmail; + private final JobConfiguration configuration; + /** * Specifies whether the job is allowed to create new tables. */ @@ -78,16 +88,44 @@ public enum WriteDisposition { WRITE_EMPTY } - private final String etag; - private final String id; - private final JobId jobId; - private final String selfLink; - private final JobStatus status; - private final JobStatistics statistics; - private final String userEmail; - private final JobConfiguration configuration; + /** + * A builder for {@code JobInfo} objects. + */ + public abstract static class Builder { + + abstract Builder etag(String etag); + + abstract Builder id(String id); + + /** + * Sets the job identity. + */ + public abstract Builder jobId(JobId jobId); + + abstract Builder selfLink(String selfLink); + + abstract Builder status(JobStatus status); + + abstract Builder statistics(JobStatistics statistics); - public static final class Builder { + abstract Builder userEmail(String userEmail); + + /** + * Sets a configuration for the {@code JobInfo} object. Use {@link CopyJobConfiguration} for a + * job that copies an existing table. Use {@link ExtractJobConfiguration} for a job that exports + * a table to Google Cloud Storage. Use {@link LoadJobConfiguration} for a job that loads data + * from Google Cloud Storage into a table. Use {@link QueryJobConfiguration} for a job that runs + * a query. + */ + public abstract Builder configuration(JobConfiguration configuration); + + /** + * Creates a {@code JobInfo} object. + */ + public abstract JobInfo build(); + } + + static final class BuilderImpl extends Builder { private String etag; private String id; @@ -98,9 +136,9 @@ public static final class Builder { private String userEmail; private JobConfiguration configuration; - private Builder() {} + BuilderImpl() {} - private Builder(JobInfo jobInfo) { + BuilderImpl(JobInfo jobInfo) { this.etag = jobInfo.etag; this.id = jobInfo.id; this.jobId = jobInfo.jobId; @@ -111,7 +149,7 @@ private Builder(JobInfo jobInfo) { this.configuration = jobInfo.configuration; } - protected Builder(Job jobPb) { + BuilderImpl(Job jobPb) { this.etag = jobPb.getEtag(); this.id = jobPb.getId(); if (jobPb.getJobReference() != null) { @@ -128,55 +166,61 @@ protected Builder(Job jobPb) { this.configuration = JobConfiguration.fromPb(jobPb.getConfiguration()); } + @Override Builder etag(String etag) { this.etag = etag; return this; } + @Override Builder id(String id) { this.id = id; return this; } - /** - * Sets the job identity. - */ + @Override public Builder jobId(JobId jobId) { this.jobId = jobId; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } + @Override Builder status(JobStatus status) { this.status = status; return this; } + @Override Builder statistics(JobStatistics statistics) { this.statistics = statistics; return this; } + @Override Builder userEmail(String userEmail) { this.userEmail = userEmail; return this; } + @Override public Builder configuration(JobConfiguration configuration) { this.configuration = configuration; return this; } + @Override public JobInfo build() { return new JobInfo(this); } } - private JobInfo(Builder builder) { + JobInfo(BuilderImpl builder) { this.jobId = builder.jobId; this.etag = builder.etag; this.id = builder.id; @@ -248,10 +292,10 @@ public C configuration() { } /** - * Returns a builder for the job. + * Returns a builder for the job object. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -275,7 +319,9 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof JobInfo && Objects.equals(toPb(), ((JobInfo) obj).toPb()); + return obj != null + && obj.getClass().equals(JobInfo.class) + && Objects.equals(toPb(), ((JobInfo) obj).toPb()); } JobInfo setProjectId(String projectId) { @@ -301,19 +347,40 @@ Job toPb() { return jobPb; } + /** + * Returns a builder for a {@code JobInfo} object given the job configuration. Use + * {@link CopyJobConfiguration} for a job that copies an existing table. Use + * {@link ExtractJobConfiguration} for a job that exports a table to Google Cloud Storage. Use + * {@link LoadJobConfiguration} for a job that loads data from Google Cloud Storage into a table. + * Use {@link QueryJobConfiguration} for a job that runs a query. + */ public static Builder builder(JobConfiguration configuration) { - return new Builder().configuration(configuration); + return new BuilderImpl().configuration(configuration); } + /** + * Returns a {@code JobInfo} object given the job configuration. Use {@link CopyJobConfiguration} + * for a job that copies an existing table. Use {@link ExtractJobConfiguration} for a job that + * exports a table to Google Cloud Storage. Use {@link LoadJobConfiguration} for a job that loads + * data from Google Cloud Storage into a table. Use {@link QueryJobConfiguration} for a job that + * runs a query. + */ public static JobInfo of(JobConfiguration configuration) { return builder(configuration).build(); } + /** + * Returns a builder for a {@code JobInfo} object given the job identity and configuration. Use + * {@link CopyJobConfiguration} for a job that copies an existing table. Use + * {@link ExtractJobConfiguration} for a job that exports a table to Google Cloud Storage. Use + * {@link LoadJobConfiguration} for a job that loads data from Google Cloud Storage into a table. + * Use {@link QueryJobConfiguration} for a job that runs a query. + */ public static JobInfo of(JobId jobId, JobConfiguration configuration) { return builder(configuration).jobId(jobId).build(); } static JobInfo fromPb(Job jobPb) { - return new Builder(jobPb).build(); + return new BuilderImpl(jobPb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java index 1f98a3dfaca1..9c9fa7a769b6 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadJobConfiguration.java @@ -282,6 +282,7 @@ LoadJobConfiguration setProjectId(String projectId) { return toBuilder().destinationTable(destinationTable().setProjectId(projectId)).build(); } + @Override com.google.api.services.bigquery.model.JobConfiguration toPb() { JobConfigurationLoad loadConfigurationPb = new JobConfigurationLoad(); loadConfigurationPb.setDestinationTable(destinationTable.toPb()); diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java index 94c16de149ed..688611d07526 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryJobConfiguration.java @@ -472,6 +472,7 @@ QueryJobConfiguration setProjectId(String projectId) { return builder.build(); } + @Override com.google.api.services.bigquery.model.JobConfiguration toPb() { com.google.api.services.bigquery.model.JobConfiguration configurationPb = new com.google.api.services.bigquery.model.JobConfiguration(); diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java index 0bcfb3d4a9ae..5f99f3c5b4ee 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryRequest.java @@ -35,26 +35,26 @@ * {@link QueryResponse#jobCompleted()} returns {@code true}. * *

Example usage of a query request: - *

    {@code
- *    // Substitute "field", "table" and "dataset" with real field, table and dataset identifiers
- *    QueryRequest request = QueryRequest.builder("SELECT field FROM table")
- *      .defaultDataset(DatasetId.of("dataset"))
- *      .maxWaitTime(60000L)
- *      .maxResults(1000L)
- *      .build();
- *    QueryResponse response = bigquery.query(request);
- *    while (!response.jobCompleted()) {
- *      Thread.sleep(1000);
- *      response = bigquery.getQueryResults(response.jobId());
- *    }
- *    List executionErrors = response.executionErrors();
- *    // look for errors in executionErrors
- *    QueryResult result = response.result();
- *    Iterator> rowIterator = result.iterateAll();
- *    while(rowIterator.hasNext()) {
- *      List row = rowIterator.next();
- *      // do something with row
- *    }
+ * 
 {@code
+ * // Substitute "field", "table" and "dataset" with real field, table and dataset identifiers
+ * QueryRequest request = QueryRequest.builder("SELECT field FROM table")
+ *     .defaultDataset(DatasetId.of("dataset"))
+ *     .maxWaitTime(60000L)
+ *     .maxResults(1000L)
+ *     .build();
+ * QueryResponse response = bigquery.query(request);
+ * while (!response.jobCompleted()) {
+ *   Thread.sleep(1000);
+ *   response = bigquery.getQueryResults(response.jobId());
+ * }
+ * List executionErrors = response.executionErrors();
+ * // look for errors in executionErrors
+ * QueryResult result = response.result();
+ * Iterator> rowIterator = result.iterateAll();
+ * while(rowIterator.hasNext()) {
+ *   List row = rowIterator.next();
+ *   // do something with row
+ * }
  * }
* * @see Query diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java index 77386747754f..12000cc1cbd2 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/QueryResponse.java @@ -29,20 +29,20 @@ * Query Request ({@link BigQuery#query(QueryRequest)}). * *

Example usage of a query response: - *

    {@code
- *    QueryResponse response = bigquery.query(request);
- *    while (!response.jobCompleted()) {
- *      Thread.sleep(1000);
- *      response = bigquery.getQueryResults(response.jobId());
- *    }
- *    List executionErrors = response.executionErrors();
- *    // look for errors in executionErrors
- *    QueryResult result = response.result();
- *    Iterator> rowIterator = result.iterateAll();
- *    while(rowIterator.hasNext()) {
- *      List row = rowIterator.next();
- *      // do something with row
- *    }
+ * 
 {@code
+ * QueryResponse response = bigquery.query(request);
+ * while (!response.jobCompleted()) {
+ *   Thread.sleep(1000);
+ *   response = bigquery.getQueryResults(response.jobId());
+ * }
+ * List executionErrors = response.executionErrors();
+ * // look for errors in executionErrors
+ * QueryResult result = response.result();
+ * Iterator> rowIterator = result.iterateAll();
+ * while(rowIterator.hasNext()) {
+ *   List row = rowIterator.next();
+ *   // do something with row
+ * }
  * }
* * @see Get Query diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java index d6e8f0176609..d0e49157a99c 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/StandardTableDefinition.java @@ -26,10 +26,11 @@ import java.util.Objects; /** - * A Google BigQuery default table type. This type is used for standard, two-dimensional tables with - * individual records organized in rows, and a data type assigned to each column (also called a - * field). Individual fields within a record may contain nested and repeated children fields. Every - * table is described by a schema that describes field names, types, and other information. + * A Google BigQuery default table definition. This definition is used for standard, two-dimensional + * tables with individual records organized in rows, and a data type assigned to each column (also + * called a field). Individual fields within a record may contain nested and repeated children + * fields. Every table is described by a schema that describes field names, types, and other + * information. * * @see Managing Tables */ @@ -218,14 +219,14 @@ public StreamingBuffer streamingBuffer() { } /** - * Returns a builder for a BigQuery default table type. + * Returns a builder for a BigQuery standard table definition. */ public static Builder builder() { return new Builder(); } /** - * Creates a BigQuery default table type given its schema. + * Creates a BigQuery standard table definition given its schema. * * @param schema the schema of the table */ diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java index cb45c52afd7e..3f902d2ff242 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/Table.java @@ -16,12 +16,13 @@ package com.google.gcloud.bigquery; -import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; +import java.io.IOException; +import java.io.ObjectInputStream; import java.util.List; import java.util.Objects; @@ -30,62 +31,106 @@ * *

Objects of this class are immutable. Operations that modify the table like {@link #update} * return a new object. To get a {@code Table} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Table} adds a layer of service-related functionality over + * {@link TableInfo}. *

*/ -public final class Table { +public final class Table extends TableInfo { - private final BigQuery bigquery; - private final TableInfo info; + private static final long serialVersionUID = 5744556727066570096L; - /** - * Constructs a {@code Table} object for the provided {@code TableInfo}. The BigQuery service - * is used to issue requests. - * - * @param bigquery the BigQuery service used for issuing requests - * @param info table's info - */ - public Table(BigQuery bigquery, TableInfo info) { - this.bigquery = checkNotNull(bigquery); - this.info = checkNotNull(info); - } + private final BigQueryOptions options; + private transient BigQuery bigquery; /** - * Creates a {@code Table} object for the provided dataset and table's user-defined ids. Performs - * an RPC call to get the latest table information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param dataset the dataset's user-defined id - * @param table the table's user-defined id - * @param options table options - * @return the {@code Table} object or {@code null} if not found - * @throws BigQueryException upon failure + * A builder for {@code Table} objects. */ - public static Table get(BigQuery bigquery, String dataset, String table, - BigQuery.TableOption... options) { - return get(bigquery, TableId.of(dataset, table), options); - } + public static class Builder extends TableInfo.Builder { - /** - * Creates a {@code Table} object for the provided table identity. Performs an RPC call to get the - * latest table information. - * - * @param bigquery the BigQuery service used for issuing requests - * @param table the table's identity - * @param options table options - * @return the {@code Table} object or {@code null} if not found - * @throws BigQueryException upon failure - */ - public static Table get(BigQuery bigquery, TableId table, BigQuery.TableOption... options) { - TableInfo info = bigquery.getTable(table, options); - return info != null ? new Table(bigquery, info) : null; + private final BigQuery bigquery; + private final TableInfo.BuilderImpl infoBuilder; + + Builder(BigQuery bigquery, TableId tableId, TableDefinition defintion) { + this.bigquery = bigquery; + this.infoBuilder = new TableInfo.BuilderImpl(); + this.infoBuilder.tableId(tableId).definition(defintion); + } + + Builder(Table table) { + this.bigquery = table.bigquery; + this.infoBuilder = new TableInfo.BuilderImpl(table); + } + + @Override + Builder creationTime(Long creationTime) { + infoBuilder.creationTime(creationTime); + return this; + } + + @Override + public Builder description(String description) { + infoBuilder.description(description); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + public Builder expirationTime(Long expirationTime) { + infoBuilder.expirationTime(expirationTime); + return this; + } + + @Override + public Builder friendlyName(String friendlyName) { + infoBuilder.friendlyName(friendlyName); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + Builder lastModifiedTime(Long lastModifiedTime) { + infoBuilder.lastModifiedTime(lastModifiedTime); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder tableId(TableId tableId) { + infoBuilder.tableId(tableId); + return this; + } + + @Override + public Builder definition(TableDefinition definition) { + infoBuilder.definition(definition); + return this; + } + + @Override + public Table build() { + return new Table(bigquery, infoBuilder); + } } - /** - * Returns the table's information. - */ - public TableInfo info() { - return info; + Table(BigQuery bigquery, TableInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.bigquery = checkNotNull(bigquery); + this.options = bigquery.options(); } /** @@ -95,7 +140,7 @@ public TableInfo info() { * @throws BigQueryException upon failure */ public boolean exists() { - return bigquery.getTable(info.tableId(), BigQuery.TableOption.fields()) != null; + return bigquery.getTable(tableId(), BigQuery.TableOption.fields()) != null; } /** @@ -106,25 +151,19 @@ public boolean exists() { * @throws BigQueryException upon failure */ public Table reload(BigQuery.TableOption... options) { - return Table.get(bigquery, info.tableId(), options); + return bigquery.getTable(tableId(), options); } /** - * Updates the table's information. Dataset's and table's user-defined ids cannot be changed. A - * new {@code Table} object is returned. + * Updates the table's information with this table's information. Dataset's and table's + * user-defined ids cannot be changed. A new {@code Table} object is returned. * - * @param tableInfo new table's information. Dataset's and table's user-defined ids must match the - * ones of the current table * @param options dataset options * @return a {@code Table} object with updated information * @throws BigQueryException upon failure */ - public Table update(TableInfo tableInfo, BigQuery.TableOption... options) { - checkArgument(Objects.equals(tableInfo.tableId().dataset(), - info.tableId().dataset()), "Dataset's user-defined ids must match"); - checkArgument(Objects.equals(tableInfo.tableId().table(), - info.tableId().table()), "Table's user-defined ids must match"); - return new Table(bigquery, bigquery.update(tableInfo, options)); + public Table update(BigQuery.TableOption... options) { + return bigquery.update(this, options); } /** @@ -134,7 +173,7 @@ public Table update(TableInfo tableInfo, BigQuery.TableOption... options) { * @throws BigQueryException upon failure */ public boolean delete() { - return bigquery.delete(info.tableId()); + return bigquery.delete(tableId()); } /** @@ -143,8 +182,9 @@ public boolean delete() { * @param rows rows to be inserted * @throws BigQueryException upon failure */ - InsertAllResponse insert(Iterable rows) throws BigQueryException { - return bigquery.insertAll(InsertAllRequest.of(info.tableId(), rows)); + public InsertAllResponse insert(Iterable rows) + throws BigQueryException { + return bigquery.insertAll(InsertAllRequest.of(tableId(), rows)); } /** @@ -158,9 +198,9 @@ InsertAllResponse insert(Iterable rows) throws Big * to be invalid * @throws BigQueryException upon failure */ - InsertAllResponse insert(Iterable rows, boolean skipInvalidRows, - boolean ignoreUnknownValues) throws BigQueryException { - InsertAllRequest request = InsertAllRequest.builder(info.tableId(), rows) + public InsertAllResponse insert(Iterable rows, + boolean skipInvalidRows, boolean ignoreUnknownValues) throws BigQueryException { + InsertAllRequest request = InsertAllRequest.builder(tableId(), rows) .skipInvalidRows(skipInvalidRows) .ignoreUnknownValues(ignoreUnknownValues) .build(); @@ -173,8 +213,9 @@ InsertAllResponse insert(Iterable rows, boolean sk * @param options table data list options * @throws BigQueryException upon failure */ - Page> list(BigQuery.TableDataListOption... options) throws BigQueryException { - return bigquery.listTableData(info.tableId(), options); + public Page> list(BigQuery.TableDataListOption... options) + throws BigQueryException { + return bigquery.listTableData(tableId(), options); } /** @@ -186,7 +227,7 @@ Page> list(BigQuery.TableDataListOption... options) throws BigQ * @param options job options * @throws BigQueryException upon failure */ - Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption... options) + public Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption... options) throws BigQueryException { return copy(TableId.of(destinationDataset, destinationTable), options); } @@ -199,9 +240,10 @@ Job copy(String destinationDataset, String destinationTable, BigQuery.JobOption. * @param options job options * @throws BigQueryException upon failure */ - Job copy(TableId destinationTable, BigQuery.JobOption... options) throws BigQueryException { - CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, info.tableId()); - return new Job(bigquery, bigquery.create(JobInfo.of(configuration), options)); + public Job copy(TableId destinationTable, BigQuery.JobOption... options) + throws BigQueryException { + CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, tableId()); + return bigquery.create(JobInfo.of(configuration), options); } /** @@ -214,7 +256,7 @@ Job copy(TableId destinationTable, BigQuery.JobOption... options) throws BigQuer * @param options job options * @throws BigQueryException upon failure */ - Job extract(String format, String destinationUri, BigQuery.JobOption... options) + public Job extract(String format, String destinationUri, BigQuery.JobOption... options) throws BigQueryException { return extract(format, ImmutableList.of(destinationUri), options); } @@ -229,11 +271,11 @@ Job extract(String format, String destinationUri, BigQuery.JobOption... options) * @param options job options * @throws BigQueryException upon failure */ - Job extract(String format, List destinationUris, BigQuery.JobOption... options) + public Job extract(String format, List destinationUris, BigQuery.JobOption... options) throws BigQueryException { ExtractJobConfiguration extractConfiguration = - ExtractJobConfiguration.of(info.tableId(), destinationUris, format); - return new Job(bigquery, bigquery.create(JobInfo.of(extractConfiguration), options)); + ExtractJobConfiguration.of(tableId(), destinationUris, format); + return bigquery.create(JobInfo.of(extractConfiguration), options); } /** @@ -246,7 +288,7 @@ Job extract(String format, List destinationUris, BigQuery.JobOption... o * @param options job options * @throws BigQueryException upon failure */ - Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) + public Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) throws BigQueryException { return load(format, ImmutableList.of(sourceUri), options); } @@ -261,10 +303,10 @@ Job load(FormatOptions format, String sourceUri, BigQuery.JobOption... options) * @param options job options * @throws BigQueryException upon failure */ - Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... options) + public Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... options) throws BigQueryException { - LoadJobConfiguration loadConfig = LoadJobConfiguration.of(info.tableId(), sourceUris, format); - return new Job(bigquery, bigquery.create(JobInfo.of(loadConfig), options)); + LoadJobConfiguration loadConfig = LoadJobConfiguration.of(tableId(), sourceUris, format); + return bigquery.create(JobInfo.of(loadConfig), options); } /** @@ -273,4 +315,30 @@ Job load(FormatOptions format, List sourceUris, BigQuery.JobOption... op public BigQuery bigquery() { return bigquery; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Table + && Objects.equals(toPb(), ((Table) obj).toPb()) + && Objects.equals(options, ((Table) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.bigquery = options.service(); + } + + static Table fromPb(BigQuery bigquery, com.google.api.services.bigquery.model.Table tablePb) { + return new Table(bigquery, new TableInfo.BuilderImpl(tablePb)); + } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java index bee0340a29a8..9c6a950ca27f 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java @@ -53,6 +53,7 @@ public void run() { } } + @Override protected StateImpl.Builder stateBuilder() { return StateImpl.builder(options(), entity(), uploadId()); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java index de858f54ac43..26e7bcc76f55 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDefinition.java @@ -25,7 +25,7 @@ import java.util.Objects; /** - * Base class for a Google BigQuery table type. + * Base class for a Google BigQuery table definition. */ public abstract class TableDefinition implements Serializable { @@ -63,10 +63,10 @@ public enum Type { } /** - * Base builder for table types. + * Base builder for table definitions. * - * @param the table type class - * @param the table type builder + * @param the table definition class + * @param the table definition builder */ public abstract static class Builder> { @@ -152,8 +152,8 @@ final int baseHashCode() { return Objects.hash(type); } - final boolean baseEquals(TableDefinition jobConfiguration) { - return Objects.equals(toPb(), jobConfiguration.toPb()); + final boolean baseEquals(TableDefinition tableDefinition) { + return Objects.equals(toPb(), tableDefinition.toPb()); } Table toPb() { diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java index 814b8cac1e97..de331350e978 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableInfo.java @@ -23,7 +23,6 @@ import com.google.api.services.bigquery.model.Table; import com.google.common.base.Function; import com.google.common.base.MoreObjects; -import com.google.common.base.MoreObjects.ToStringHelper; import java.io.Serializable; import java.math.BigInteger; @@ -36,7 +35,7 @@ * * @see Managing Tables */ -public final class TableInfo implements Serializable { +public class TableInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -67,9 +66,55 @@ public Table apply(TableInfo tableInfo) { private final TableDefinition definition; /** - * Builder for tables. + * A builder for {@code TableInfo} objects. */ - public static class Builder { + public abstract static class Builder { + + abstract Builder creationTime(Long creationTime); + + /** + * Sets a user-friendly description for the table. + */ + public abstract Builder description(String description); + + abstract Builder etag(String etag); + + /** + * Sets the time when this table expires, in milliseconds since the epoch. If not present, the + * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. + */ + public abstract Builder expirationTime(Long expirationTime); + + /** + * Sets a user-friendly name for the table. + */ + public abstract Builder friendlyName(String friendlyName); + + abstract Builder id(String id); + + abstract Builder lastModifiedTime(Long lastModifiedTime); + + abstract Builder selfLink(String selfLink); + + /** + * Sets the table identity. + */ + public abstract Builder tableId(TableId tableId); + + /** + * Sets the table definition. Use {@link StandardTableDefinition} to create simple BigQuery + * table. Use {@link ViewDefinition} to create a BigQuery view. Use + * {@link ExternalTableDefinition} to create a BigQuery a table backed by external data. + */ + public abstract Builder definition(TableDefinition definition); + + /** + * Creates a {@code TableInfo} object. + */ + public abstract TableInfo build(); + } + + static class BuilderImpl extends Builder { private String etag; private String id; @@ -82,9 +127,9 @@ public static class Builder { private Long lastModifiedTime; private TableDefinition definition; - private Builder() {} + BuilderImpl() {} - private Builder(TableInfo tableInfo) { + BuilderImpl(TableInfo tableInfo) { this.etag = tableInfo.etag; this.id = tableInfo.id; this.selfLink = tableInfo.selfLink; @@ -97,7 +142,7 @@ private Builder(TableInfo tableInfo) { this.definition = tableInfo.definition; } - private Builder(Table tablePb) { + BuilderImpl(Table tablePb) { this.tableId = TableId.fromPb(tablePb.getTableReference()); if (tablePb.getLastModifiedTime() != null) { this.lastModifiedTime(tablePb.getLastModifiedTime().longValue()); @@ -112,83 +157,73 @@ private Builder(Table tablePb) { this.definition = TableDefinition.fromPb(tablePb); } + @Override Builder creationTime(Long creationTime) { this.creationTime = creationTime; return this; } - /** - * Sets a user-friendly description for the table. - */ + @Override public Builder description(String description) { this.description = firstNonNull(description, Data.nullOf(String.class)); return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } - /** - * Sets the time when this table expires, in milliseconds since the epoch. If not present, the - * table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. - */ + @Override public Builder expirationTime(Long expirationTime) { this.expirationTime = firstNonNull(expirationTime, Data.nullOf(Long.class)); return this; } - /** - * Sets a user-friendly name for the table. - */ + @Override public Builder friendlyName(String friendlyName) { this.friendlyName = firstNonNull(friendlyName, Data.nullOf(String.class)); return this; } + @Override Builder id(String id) { this.id = id; return this; } + @Override Builder lastModifiedTime(Long lastModifiedTime) { this.lastModifiedTime = lastModifiedTime; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Sets the table identity. - */ + @Override public Builder tableId(TableId tableId) { this.tableId = checkNotNull(tableId); return this; } - /** - * Sets the table definition. Use {@link StandardTableDefinition} to create simple BigQuery - * table. Use {@link ViewDefinition} to create a BigQuery view. Use - * {@link ExternalTableDefinition} to create a BigQuery a table backed by external data. - */ + @Override public Builder definition(TableDefinition definition) { this.definition = checkNotNull(definition); return this; } - /** - * Creates a {@code TableInfo} object. - */ + @Override public TableInfo build() { return new TableInfo(this); } } - private TableInfo(Builder builder) { + TableInfo(BuilderImpl builder) { this.tableId = checkNotNull(builder.tableId); this.etag = builder.etag; this.id = builder.id; @@ -275,13 +310,14 @@ public T definition() { } /** - * Returns a builder for the object. + * Returns a builder for the table object. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } - ToStringHelper toStringHelper() { + @Override + public String toString() { return MoreObjects.toStringHelper(this) .add("tableId", tableId) .add("etag", etag) @@ -292,12 +328,8 @@ ToStringHelper toStringHelper() { .add("expirationTime", expirationTime) .add("creationTime", creationTime) .add("lastModifiedTime", lastModifiedTime) - .add("definition", definition); - } - - @Override - public String toString() { - return toStringHelper().toString(); + .add("definition", definition) + .toString(); } @Override @@ -307,18 +339,26 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof TableInfo && Objects.equals(toPb(), ((TableInfo) obj).toPb()); + return obj != null + && obj.getClass().equals(TableInfo.class) + && Objects.equals(toPb(), ((TableInfo) obj).toPb()); } /** - * Returns a builder for a {@code TableInfo} object given table identity and definition. + * Returns a builder for a {@code TableInfo} object given table identity and definition. Use + * {@link StandardTableDefinition} to create simple BigQuery table. Use {@link ViewDefinition} to + * create a BigQuery view. Use {@link ExternalTableDefinition} to create a BigQuery a table backed + * by external data. */ public static Builder builder(TableId tableId, TableDefinition definition) { - return new Builder().tableId(tableId).definition(definition); + return new BuilderImpl().tableId(tableId).definition(definition); } /** - * Returns a {@code TableInfo} object given table identity and definition. + * Returns a {@code TableInfo} object given table identity and definition. Use + * {@link StandardTableDefinition} to create simple BigQuery table. Use {@link ViewDefinition} to + * create a BigQuery view. Use {@link ExternalTableDefinition} to create a BigQuery a table backed + * by external data. */ public static TableInfo of(TableId tableId, TableDefinition definition) { return builder(tableId, definition).build(); @@ -345,6 +385,6 @@ Table toPb() { } static TableInfo fromPb(Table tablePb) { - return new Builder(tablePb).build(); + return new BuilderImpl(tablePb).build(); } } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java index 7065bcc155d2..796dd411b4a1 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/ViewDefinition.java @@ -27,8 +27,9 @@ import java.util.Objects; /** - * Google BigQuery view table type. BigQuery's views are logical views, not materialized views, - * which means that the query that defines the view is re-executed every time the view is queried. + * Google BigQuery view table definition. BigQuery's views are logical views, not materialized + * views, which means that the query that defines the view is re-executed every time the view is + * queried. * * @see Views */ @@ -168,7 +169,7 @@ Table toPb() { } /** - * Returns a builder for a BigQuery view type. + * Returns a builder for a BigQuery view definition. * * @param query the query used to generate the view */ @@ -177,7 +178,7 @@ public static Builder builder(String query) { } /** - * Returns a builder for a BigQuery view type. + * Returns a builder for a BigQuery view definition. * * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query @@ -187,7 +188,7 @@ public static Builder builder(String query, List functions) } /** - * Returns a builder for a BigQuery view type. + * Returns a builder for a BigQuery view definition. * * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query @@ -197,7 +198,7 @@ public static Builder builder(String query, UserDefinedFunction... functions) { } /** - * Creates a BigQuery view type given the query used to generate the table. + * Creates a BigQuery view definition given the query used to generate the table. * * @param query the query used to generate the table */ @@ -206,7 +207,7 @@ public static ViewDefinition of(String query) { } /** - * Creates a BigQuery view type given a query and some user-defined functions. + * Creates a BigQuery view definition given a query and some user-defined functions. * * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query @@ -216,7 +217,7 @@ public static ViewDefinition of(String query, List function } /** - * Creates a BigQuery view type given a query and some user-defined functions. + * Creates a BigQuery view definition given a query and some user-defined functions. * * @param query the query used to generate the table * @param functions user-defined functions that can be used by the query diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java index 18342bac1bff..6cc44ce7d5d6 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/WriteChannelConfiguration.java @@ -155,6 +155,7 @@ public Builder projectionFields(List projectionFields) { return this; } + @Override public WriteChannelConfiguration build() { return new WriteChannelConfiguration(this); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java index a249768f9d8d..db5e956e0a12 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/package-info.java @@ -17,30 +17,29 @@ /** * A client to Google Cloud BigQuery. * - *

A simple usage example: + *

A simple usage example showing how to create a table if it does not exist and load data into + * it. For the complete source code see + * + * CreateTableAndLoadData.java. *

 {@code
  * BigQuery bigquery = BigQueryOptions.defaultInstance().service();
  * TableId tableId = TableId.of("dataset", "table");
- * TableInfo info = bigquery.getTable(tableId);
- * if (info == null) {
+ * Table table = bigquery.getTable(tableId);
+ * if (table == null) {
  *   System.out.println("Creating table " + tableId);
  *   Field integerField = Field.of("fieldName", Field.Type.integer());
  *   Schema schema = Schema.of(integerField);
- *   bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema)));
+ *   table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema)));
+ * }
+ * System.out.println("Loading data into table " + tableId);
+ * Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path");
+ * while (!loadJob.isDone()) {
+ *   Thread.sleep(1000L);
+ * }
+ * if (loadJob.status().error() != null) {
+ *   System.out.println("Job completed with errors");
  * } else {
- *   System.out.println("Loading data into table " + tableId);
- *   LoadJobConfiguration configuration = LoadJobConfiguration.of(tableId, "gs://bucket/path");
- *   JobInfo loadJob = JobInfo.of(configuration);
- *   loadJob = bigquery.create(loadJob);
- *   while (loadJob.status().state() != JobStatus.State.DONE) {
- *     Thread.sleep(1000L);
- *     loadJob = bigquery.getJob(loadJob.jobId());
- *   }
- *   if (loadJob.status().error() != null) {
- *     System.out.println("Job completed with errors");
- *   } else {
- *     System.out.println("Job succeeded");
- *   }
+ *   System.out.println("Job succeeded");
  * }}
* * @see Google Cloud BigQuery diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java index 6062e19950e0..a1935e5ab136 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/BigQueryRpc.java @@ -100,7 +100,7 @@ public Y y() { * * @throws BigQueryException upon failure */ - Dataset getDataset(String datasetId, Map options) throws BigQueryException; + Dataset getDataset(String datasetId, Map options); /** * Lists the project's datasets. Partial information is returned on a dataset (datasetReference, @@ -108,13 +108,28 @@ public Y y() { * * @throws BigQueryException upon failure */ - Tuple> listDatasets(Map options) throws BigQueryException; + Tuple> listDatasets(Map options); - Dataset create(Dataset dataset, Map options) throws BigQueryException; + /** + * Creates a new dataset. + * + * @throws BigQueryException upon failure + */ + Dataset create(Dataset dataset, Map options); - Table create(Table table, Map options) throws BigQueryException; + /** + * Creates a new table. + * + * @throws BigQueryException upon failure + */ + Table create(Table table, Map options); - Job create(Job job, Map options) throws BigQueryException; + /** + * Creates a new job. + * + * @throws BigQueryException upon failure + */ + Job create(Job job, Map options); /** * Delete the requested dataset. @@ -122,18 +137,28 @@ public Y y() { * @return {@code true} if dataset was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean deleteDataset(String datasetId, Map options) throws BigQueryException; + boolean deleteDataset(String datasetId, Map options); - Dataset patch(Dataset dataset, Map options) throws BigQueryException; + /** + * Updates dataset information. + * + * @throws BigQueryException upon failure + */ + Dataset patch(Dataset dataset, Map options); - Table patch(Table table, Map options) throws BigQueryException; + /** + * Updates table information. + * + * @throws BigQueryException upon failure + */ + Table patch(Table table, Map options); /** * Returns the requested table or {@code null} if not found. * * @throws BigQueryException upon failure */ - Table getTable(String datasetId, String tableId, Map options) throws BigQueryException; + Table getTable(String datasetId, String tableId, Map options); /** * Lists the dataset's tables. Partial information is returned on a table (tableReference, @@ -141,8 +166,7 @@ public Y y() { * * @throws BigQueryException upon failure */ - Tuple> listTables(String dataset, Map options) - throws BigQueryException; + Tuple> listTables(String dataset, Map options); /** * Delete the requested table. @@ -150,27 +174,37 @@ Tuple> listTables(String dataset, Map options * @return {@code true} if table was deleted, {@code false} if it was not found * @throws BigQueryException upon failure */ - boolean deleteTable(String datasetId, String tableId) throws BigQueryException; + boolean deleteTable(String datasetId, String tableId); + /** + * Sends an insert all request. + * + * @throws BigQueryException upon failure + */ TableDataInsertAllResponse insertAll(String datasetId, String tableId, - TableDataInsertAllRequest request) throws BigQueryException; + TableDataInsertAllRequest request); + /** + * Lists the table's rows. + * + * @throws BigQueryException upon failure + */ Tuple> listTableData(String datasetId, String tableId, - Map options) throws BigQueryException; + Map options); /** * Returns the requested job or {@code null} if not found. * * @throws BigQueryException upon failure */ - Job getJob(String jobId, Map options) throws BigQueryException; + Job getJob(String jobId, Map options); /** * Lists the project's jobs. * * @throws BigQueryException upon failure */ - Tuple> listJobs(Map options) throws BigQueryException; + Tuple> listJobs(Map options); /** * Sends a job cancel request. This call will return immediately, and the client will need to poll @@ -180,12 +214,21 @@ Tuple> listTableData(String datasetId, String tableId * found * @throws BigQueryException upon failure */ - boolean cancel(String jobId) throws BigQueryException; + boolean cancel(String jobId); - GetQueryResultsResponse getQueryResults(String jobId, Map options) - throws BigQueryException; + /** + * Returns results of the query associated with the provided job. + * + * @throws BigQueryException upon failure + */ + GetQueryResultsResponse getQueryResults(String jobId, Map options); - QueryResponse query(QueryRequest request) throws BigQueryException; + /** + * Runs the query associated with the request. + * + * @throws BigQueryException upon failure + */ + QueryResponse query(QueryRequest request); /** * Opens a resumable upload session to load data into a BigQuery table and returns an upload URI. @@ -193,7 +236,7 @@ GetQueryResultsResponse getQueryResults(String jobId, Map options) * @param configuration load configuration * @throws BigQueryException upon failure */ - String open(JobConfiguration configuration) throws BigQueryException; + String open(JobConfiguration configuration); /** * Uploads the provided data to the resumable upload session at the specified position. @@ -207,5 +250,5 @@ GetQueryResultsResponse getQueryResults(String jobId, Map options) * @throws BigQueryException upon failure */ void write(String uploadId, byte[] toWrite, int toWriteOffset, long destOffset, int length, - boolean last) throws BigQueryException; + boolean last); } diff --git a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java index b57f1dc8a128..a5c44129955a 100644 --- a/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java +++ b/gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java @@ -90,7 +90,7 @@ private static BigQueryException translate(IOException exception) { } @Override - public Dataset getDataset(String datasetId, Map options) throws BigQueryException { + public Dataset getDataset(String datasetId, Map options) { try { return bigquery.datasets() .get(this.options.projectId(), datasetId) @@ -106,8 +106,7 @@ public Dataset getDataset(String datasetId, Map options) throws BigQu } @Override - public Tuple> listDatasets(Map options) - throws BigQueryException { + public Tuple> listDatasets(Map options) { try { DatasetList datasetsList = bigquery.datasets() .list(this.options.projectId()) @@ -135,7 +134,7 @@ public Dataset apply(DatasetList.Datasets datasetPb) { } @Override - public Dataset create(Dataset dataset, Map options) throws BigQueryException { + public Dataset create(Dataset dataset, Map options) { try { return bigquery.datasets().insert(this.options.projectId(), dataset) .setFields(FIELDS.getString(options)) @@ -146,8 +145,7 @@ public Dataset create(Dataset dataset, Map options) throws BigQueryEx } @Override - public Table create(Table table, Map options) - throws BigQueryException { + public Table create(Table table, Map options) { try { // unset the type, as it is output only table.setType(null); @@ -161,7 +159,7 @@ public Table create(Table table, Map options) } @Override - public Job create(Job job, Map options) throws BigQueryException { + public Job create(Job job, Map options) { try { return bigquery.jobs() .insert(this.options.projectId(), job) @@ -173,7 +171,7 @@ public Job create(Job job, Map options) throws BigQueryException { } @Override - public boolean deleteDataset(String datasetId, Map options) throws BigQueryException { + public boolean deleteDataset(String datasetId, Map options) { try { bigquery.datasets().delete(this.options.projectId(), datasetId) .setDeleteContents(DELETE_CONTENTS.getBoolean(options)) @@ -189,7 +187,7 @@ public boolean deleteDataset(String datasetId, Map options) throws Bi } @Override - public Dataset patch(Dataset dataset, Map options) throws BigQueryException { + public Dataset patch(Dataset dataset, Map options) { try { DatasetReference reference = dataset.getDatasetReference(); return bigquery.datasets() @@ -202,7 +200,7 @@ public Dataset patch(Dataset dataset, Map options) throws BigQueryExc } @Override - public Table patch(Table table, Map options) throws BigQueryException { + public Table patch(Table table, Map options) { try { // unset the type, as it is output only table.setType(null); @@ -217,8 +215,7 @@ public Table patch(Table table, Map options) throws BigQueryException } @Override - public Table getTable(String datasetId, String tableId, Map options) - throws BigQueryException { + public Table getTable(String datasetId, String tableId, Map options) { try { return bigquery.tables() .get(this.options.projectId(), datasetId, tableId) @@ -234,8 +231,7 @@ public Table getTable(String datasetId, String tableId, Map options) } @Override - public Tuple> listTables(String datasetId, Map options) - throws BigQueryException { + public Tuple> listTables(String datasetId, Map options) { try { TableList tableList = bigquery.tables() .list(this.options.projectId(), datasetId) @@ -262,7 +258,7 @@ public Table apply(TableList.Tables tablePb) { } @Override - public boolean deleteTable(String datasetId, String tableId) throws BigQueryException { + public boolean deleteTable(String datasetId, String tableId) { try { bigquery.tables().delete(this.options.projectId(), datasetId, tableId).execute(); return true; @@ -277,7 +273,7 @@ public boolean deleteTable(String datasetId, String tableId) throws BigQueryExce @Override public TableDataInsertAllResponse insertAll(String datasetId, String tableId, - TableDataInsertAllRequest request) throws BigQueryException { + TableDataInsertAllRequest request) { try { return bigquery.tabledata() .insertAll(this.options.projectId(), datasetId, tableId, request) @@ -289,7 +285,7 @@ public TableDataInsertAllResponse insertAll(String datasetId, String tableId, @Override public Tuple> listTableData(String datasetId, String tableId, - Map options) throws BigQueryException { + Map options) { try { TableDataList tableDataList = bigquery.tabledata() .list(this.options.projectId(), datasetId, tableId) @@ -306,7 +302,7 @@ public Tuple> listTableData(String datasetId, String } @Override - public Job getJob(String jobId, Map options) throws BigQueryException { + public Job getJob(String jobId, Map options) { try { return bigquery.jobs() .get(this.options.projectId(), jobId) @@ -322,7 +318,7 @@ public Job getJob(String jobId, Map options) throws BigQueryException } @Override - public Tuple> listJobs(Map options) throws BigQueryException { + public Tuple> listJobs(Map options) { try { JobList jobsList = bigquery.jobs() .list(this.options.projectId()) @@ -363,7 +359,7 @@ public Job apply(JobList.Jobs jobPb) { } @Override - public boolean cancel(String jobId) throws BigQueryException { + public boolean cancel(String jobId) { try { bigquery.jobs().cancel(this.options.projectId(), jobId).execute(); return true; @@ -377,8 +373,7 @@ public boolean cancel(String jobId) throws BigQueryException { } @Override - public GetQueryResultsResponse getQueryResults(String jobId, Map options) - throws BigQueryException { + public GetQueryResultsResponse getQueryResults(String jobId, Map options) { try { return bigquery.jobs().getQueryResults(this.options.projectId(), jobId) .setMaxResults(MAX_RESULTS.getLong(options)) @@ -397,7 +392,7 @@ public GetQueryResultsResponse getQueryResults(String jobId, Map opti } @Override - public QueryResponse query(QueryRequest request) throws BigQueryException { + public QueryResponse query(QueryRequest request) { try { return bigquery.jobs().query(this.options.projectId(), request).execute(); } catch (IOException ex) { @@ -406,7 +401,7 @@ public QueryResponse query(QueryRequest request) throws BigQueryException { } @Override - public String open(JobConfiguration configuration) throws BigQueryException { + public String open(JobConfiguration configuration) { try { Job loadJob = new Job().setConfiguration(configuration); StringBuilder builder = new StringBuilder() @@ -429,7 +424,7 @@ public String open(JobConfiguration configuration) throws BigQueryException { @Override public void write(String uploadId, byte[] toWrite, int toWriteOffset, long destOffset, int length, - boolean last) throws BigQueryException { + boolean last) { try { GenericUrl url = new GenericUrl(uploadId); HttpRequest httpRequest = bigquery.getRequestFactory().buildPutRequest(url, diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java index afad9041e802..385ee6dcc8bd 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java @@ -26,10 +26,7 @@ import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; -import com.google.api.services.bigquery.model.Dataset; import com.google.api.services.bigquery.model.ErrorProto; -import com.google.api.services.bigquery.model.Job; -import com.google.api.services.bigquery.model.Table; import com.google.api.services.bigquery.model.TableCell; import com.google.api.services.bigquery.model.TableDataInsertAllRequest; import com.google.api.services.bigquery.model.TableDataInsertAllResponse; @@ -287,8 +284,9 @@ public void testCreateDataset() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.create(DATASET_INFO); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.create(DATASET_INFO); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -299,13 +297,14 @@ public void testCreateDatasetWithSelectedFields() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.create(DATASET_INFO, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.create(DATASET_INFO, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -314,8 +313,9 @@ public void testGetDataset() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DATASET); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DATASET); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -324,8 +324,9 @@ public void testGetDatasetFromDatasetId() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DatasetId.of(PROJECT, DATASET)); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DatasetId.of(PROJECT, DATASET)); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test @@ -335,54 +336,58 @@ public void testGetDatasetWithSelectedFields() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.getDataset(DATASET, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.getDataset(DATASET, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test public void testListDatasets() { String cursor = "cursor"; - ImmutableList datasetList = ImmutableList.of(DATASET_INFO_WITH_PROJECT, - OTHER_DATASET_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList datasetList = ImmutableList.of( + new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + new Dataset(bigquery, new DatasetInfo.BuilderImpl(OTHER_DATASET_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(datasetList, DatasetInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listDatasets(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listDatasets(); + Page page = bigquery.listDatasets(); assertEquals(cursor, page.nextPageCursor()); assertArrayEquals(datasetList.toArray(), Iterables.toArray(page.values(), DatasetInfo.class)); } @Test public void testListEmptyDatasets() { - ImmutableList datasets = ImmutableList.of(); - Tuple> result = Tuple.>of(null, datasets); + ImmutableList datasets = ImmutableList.of(); + Tuple> result = + Tuple.>of(null, datasets); EasyMock.expect(bigqueryRpcMock.listDatasets(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - Page page = bigquery.listDatasets(); + Page page = bigquery.listDatasets(); assertNull(page.nextPageCursor()); assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), DatasetInfo.class)); + Iterables.toArray(page.values(), Dataset.class)); } @Test public void testListDatasetsWithOptions() { String cursor = "cursor"; - ImmutableList datasetList = ImmutableList.of(DATASET_INFO_WITH_PROJECT, - OTHER_DATASET_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList datasetList = ImmutableList.of( + new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + new Dataset(bigquery, new DatasetInfo.BuilderImpl(OTHER_DATASET_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(datasetList, DatasetInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listDatasets(DATASET_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listDatasets(DATASET_LIST_ALL, DATASET_LIST_PAGE_TOKEN, + Page page = bigquery.listDatasets(DATASET_LIST_ALL, DATASET_LIST_PAGE_TOKEN, DATASET_LIST_MAX_RESULTS); assertEquals(cursor, page.nextPageCursor()); assertArrayEquals(datasetList.toArray(), Iterables.toArray(page.values(), DatasetInfo.class)); @@ -422,8 +427,9 @@ public void testUpdateDataset() { .andReturn(updatedDatasetInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.update(updatedDatasetInfo); - assertEquals(updatedDatasetInfoWithProject, dataset); + Dataset dataset = bigquery.update(updatedDatasetInfo); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(updatedDatasetInfoWithProject)), + dataset); } @Test @@ -438,13 +444,14 @@ public void testUpdateDatasetWithSelectedFields() { .andReturn(updatedDatasetInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - DatasetInfo dataset = bigquery.update(updatedDatasetInfo, DATASET_OPTION_FIELDS); + Dataset dataset = bigquery.update(updatedDatasetInfo, DATASET_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(DATASET_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("datasetReference")); assertTrue(selector.contains("access")); assertTrue(selector.contains("etag")); assertEquals(28, selector.length()); - assertEquals(updatedDatasetInfoWithProject, dataset); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(updatedDatasetInfoWithProject)), + dataset); } @Test @@ -453,8 +460,8 @@ public void testCreateTable() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.create(TABLE_INFO); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.create(TABLE_INFO); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -465,13 +472,13 @@ public void testCreateTableWithSelectedFields() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.create(TABLE_INFO, TABLE_OPTION_FIELDS); + Table table = bigquery.create(TABLE_INFO, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -480,8 +487,8 @@ public void testGetTable() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(DATASET, TABLE); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.getTable(DATASET, TABLE); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -490,8 +497,8 @@ public void testGetTableFromTableId() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(TABLE_ID); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + Table table = bigquery.getTable(TABLE_ID); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test @@ -501,59 +508,61 @@ public void testGetTableWithSelectedFields() { .andReturn(TABLE_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.getTable(TABLE_ID, TABLE_OPTION_FIELDS); + Table table = bigquery.getTable(TABLE_ID, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(TABLE_INFO_WITH_PROJECT, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), table); } @Test public void testListTables() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DATASET); + Page
page = bigquery.listTables(DATASET); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), TableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test public void testListTablesFromDatasetId() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DatasetId.of(PROJECT, DATASET)); + Page
page = bigquery.listTables(DatasetId.of(PROJECT, DATASET)); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), TableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test public void testListTablesWithOptions() { String cursor = "cursor"; - ImmutableList tableList = - ImmutableList.of(TABLE_INFO_WITH_PROJECT, OTHER_TABLE_INFO); - Tuple> result = + bigquery = options.service(); + ImmutableList
tableList = ImmutableList.of( + new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO_WITH_PROJECT)), + new Table(bigquery, new TableInfo.BuilderImpl(OTHER_TABLE_INFO))); + Tuple> result = Tuple.of(cursor, Iterables.transform(tableList, TableInfo.TO_PB_FUNCTION)); EasyMock.expect(bigqueryRpcMock.listTables(DATASET, TABLE_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listTables(DATASET, TABLE_LIST_MAX_RESULTS, - TABLE_LIST_PAGE_TOKEN); + Page
page = bigquery.listTables(DATASET, TABLE_LIST_MAX_RESULTS, TABLE_LIST_PAGE_TOKEN); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), TableInfo.class)); + assertArrayEquals(tableList.toArray(), Iterables.toArray(page.values(), Table.class)); } @Test @@ -582,8 +591,9 @@ public void testUpdateTable() { .andReturn(updatedTableInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.update(updatedTableInfo); - assertEquals(updatedTableInfoWithProject, table); + Table table = bigquery.update(updatedTableInfo); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(updatedTableInfoWithProject)), + table); } @Test @@ -597,13 +607,14 @@ public void testUpdateTableWithSelectedFields() { capture(capturedOptions))).andReturn(updatedTableInfoWithProject.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - TableInfo table = bigquery.update(updatedTableInfo, TABLE_OPTION_FIELDS); + Table table = bigquery.update(updatedTableInfo, TABLE_OPTION_FIELDS); String selector = (String) capturedOptions.getValue().get(TABLE_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("tableReference")); assertTrue(selector.contains("schema")); assertTrue(selector.contains("etag")); assertEquals(31, selector.length()); - assertEquals(updatedTableInfoWithProject, table); + assertEquals(new Table(bigquery, new TableInfo.BuilderImpl(updatedTableInfoWithProject)), + table); } @Test @@ -627,8 +638,7 @@ public TableDataInsertAllRequest.Rows apply(RowToInsert rowToInsert) { return new TableDataInsertAllRequest.Rows().setInsertId(rowToInsert.id()) .setJson(rowToInsert.content()); } - }) - ).setSkipInvalidRows(false).setIgnoreUnknownValues(true).setTemplateSuffix("suffix"); + })).setSkipInvalidRows(false).setIgnoreUnknownValues(true).setTemplateSuffix("suffix"); TableDataInsertAllResponse responsePb = new TableDataInsertAllResponse().setInsertErrors( ImmutableList.of(new TableDataInsertAllResponse.InsertErrors().setIndex(0L).setErrors( ImmutableList.of(new ErrorProto().setMessage("ErrorMessage"))))); @@ -731,57 +741,57 @@ public void testListTableDataWithOptions() { @Test public void testCreateQueryJob() { EasyMock.expect(bigqueryRpcMock.create( - JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_QUERY_JOB.toPb()); + JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_QUERY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.create(QUERY_JOB); - assertEquals(COMPLETE_QUERY_JOB, job); + Job job = bigquery.create(QUERY_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), job); } @Test public void testCreateLoadJob() { EasyMock.expect(bigqueryRpcMock.create( - JobInfo.of(LOAD_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_LOAD_JOB.toPb()); + JobInfo.of(LOAD_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_LOAD_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.create(LOAD_JOB); - assertEquals(COMPLETE_LOAD_JOB, job); + Job job = bigquery.create(LOAD_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB)), job); } @Test public void testCreateCopyJob() { EasyMock.expect(bigqueryRpcMock.create( - JobInfo.of(COPY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_COPY_JOB.toPb()); + JobInfo.of(COPY_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.create(COPY_JOB); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.create(COPY_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test public void testCreateExtractJob() { EasyMock.expect(bigqueryRpcMock.create( - JobInfo.of(EXTRACT_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) - .andReturn(COMPLETE_EXTRACT_JOB.toPb()); + JobInfo.of(EXTRACT_JOB_CONFIGURATION_WITH_PROJECT).toPb(), EMPTY_RPC_OPTIONS)) + .andReturn(COMPLETE_EXTRACT_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.create(EXTRACT_JOB); - assertEquals(COMPLETE_EXTRACT_JOB, job); + Job job = bigquery.create(EXTRACT_JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_EXTRACT_JOB)), job); } @Test public void testCreateJobWithSelectedFields() { Capture> capturedOptions = Capture.newInstance(); EasyMock.expect(bigqueryRpcMock.create( - eq(JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb()), capture(capturedOptions))) - .andReturn(COMPLETE_QUERY_JOB.toPb()); + eq(JobInfo.of(QUERY_JOB_CONFIGURATION_WITH_PROJECT).toPb()), capture(capturedOptions))) + .andReturn(COMPLETE_QUERY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.create(QUERY_JOB, JOB_OPTION_FIELDS); - assertEquals(COMPLETE_QUERY_JOB, job); + Job job = bigquery.create(QUERY_JOB, JOB_OPTION_FIELDS); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), job); String selector = (String) capturedOptions.getValue().get(JOB_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("jobReference")); assertTrue(selector.contains("configuration")); @@ -795,8 +805,8 @@ public void testGetJob() { .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.getJob(JOB); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.getJob(JOB); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test @@ -805,67 +815,76 @@ public void testGetJobFromJobId() { .andReturn(COMPLETE_COPY_JOB.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.service(); - JobInfo job = bigquery.getJob(JobId.of(PROJECT, JOB)); - assertEquals(COMPLETE_COPY_JOB, job); + Job job = bigquery.getJob(JobId.of(PROJECT, JOB)); + assertEquals(new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_COPY_JOB)), job); } @Test public void testListJobs() { String cursor = "cursor"; - ImmutableList jobList = ImmutableList.of(COMPLETE_QUERY_JOB, COMPLETE_LOAD_JOB); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(); + Page page = bigquery.listJobs(); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); } @Test public void testListJobsWithOptions() { String cursor = "cursor"; - ImmutableList jobList = ImmutableList.of(COMPLETE_QUERY_JOB, COMPLETE_LOAD_JOB); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(JOB_LIST_OPTIONS)).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(JOB_LIST_ALL_USERS, JOB_LIST_STATE_FILTER, + Page page = bigquery.listJobs(JOB_LIST_ALL_USERS, JOB_LIST_STATE_FILTER, JOB_LIST_PAGE_TOKEN, JOB_LIST_MAX_RESULTS); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); } @Test public void testListJobsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList jobList = ImmutableList.of(COMPLETE_QUERY_JOB, COMPLETE_LOAD_JOB); - Tuple> result = - Tuple.of(cursor, Iterables.transform(jobList, new Function() { - @Override - public Job apply(JobInfo jobInfo) { - return jobInfo.toPb(); - } - })); + bigquery = options.service(); + ImmutableList jobList = ImmutableList.of( + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_QUERY_JOB)), + new Job(bigquery, new JobInfo.BuilderImpl(COMPLETE_LOAD_JOB))); + Tuple> result = + Tuple.of(cursor, Iterables.transform(jobList, + new Function() { + @Override + public com.google.api.services.bigquery.model.Job apply(Job job) { + return job.toPb(); + } + })); EasyMock.expect(bigqueryRpcMock.listJobs(capture(capturedOptions))).andReturn(result); EasyMock.replay(bigqueryRpcMock); - bigquery = options.service(); - Page page = bigquery.listJobs(JOB_LIST_OPTION_FIELD); + Page page = bigquery.listJobs(JOB_LIST_OPTION_FIELD); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), JobInfo.class)); + assertArrayEquals(jobList.toArray(), Iterables.toArray(page.values(), Job.class)); String selector = (String) capturedOptions.getValue().get(JOB_OPTION_FIELDS.rpcOption()); assertTrue(selector.contains("etag,jobs(")); assertTrue(selector.contains("configuration")); @@ -1030,8 +1049,9 @@ public void testRetryableException() { .andReturn(DATASET_INFO_WITH_PROJECT.toPb()); EasyMock.replay(bigqueryRpcMock); bigquery = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); - DatasetInfo dataset = bigquery.getDataset(DATASET); - assertEquals(DATASET_INFO_WITH_PROJECT, dataset); + Dataset dataset = bigquery.getDataset(DATASET); + assertEquals(new Dataset(bigquery, new DatasetInfo.BuilderImpl(DATASET_INFO_WITH_PROJECT)), + dataset); } @Test diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java index 455212e16d3a..373291021b23 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/DatasetTest.java @@ -16,10 +16,13 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; +import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; @@ -28,20 +31,30 @@ import static org.junit.Assert.assertTrue; import com.google.common.collect.ImmutableList; +import com.google.common.collect.Iterables; import com.google.gcloud.Page; import com.google.gcloud.PageImpl; import org.junit.After; -import org.junit.Before; -import org.junit.Rule; import org.junit.Test; -import org.junit.rules.ExpectedException; -import java.util.Iterator; +import java.util.List; public class DatasetTest { private static final DatasetId DATASET_ID = DatasetId.of("dataset"); + private static final List ACCESS_RULES = ImmutableList.of( + Acl.of(Acl.Group.ofAllAuthenticatedUsers(), Acl.Role.READER), + Acl.of(new Acl.View(TableId.of("dataset", "table")))); + private static final Long CREATION_TIME = System.currentTimeMillis(); + private static final Long DEFAULT_TABLE_EXPIRATION = CREATION_TIME + 100; + private static final String DESCRIPTION = "description"; + private static final String ETAG = "0xFF00"; + private static final String FRIENDLY_NAME = "friendlyDataset"; + private static final String ID = "P/D:1"; + private static final Long LAST_MODIFIED = CREATION_TIME + 50; + private static final String LOCATION = ""; + private static final String SELF_LINK = "http://bigquery/p/d"; private static final DatasetInfo DATASET_INFO = DatasetInfo.builder(DATASET_ID).build(); private static final Field FIELD = Field.of("FieldName", Field.Type.integer()); private static final StandardTableDefinition TABLE_DEFINITION = @@ -49,232 +62,312 @@ public class DatasetTest { private static final ViewDefinition VIEW_DEFINITION = ViewDefinition.of("QUERY"); private static final ExternalTableDefinition EXTERNAL_TABLE_DEFINITION = ExternalTableDefinition.of(ImmutableList.of("URI"), Schema.of(), FormatOptions.csv()); - private static final Iterable TABLE_INFO_RESULTS = ImmutableList.of( - TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(), - TableInfo.builder(TableId.of("dataset", "table2"), VIEW_DEFINITION).build(), - TableInfo.builder(TableId.of("dataset", "table2"), EXTERNAL_TABLE_DEFINITION).build()); + private static final TableInfo TABLE_INFO1 = + TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); + private static final TableInfo TABLE_INFO2 = + TableInfo.builder(TableId.of("dataset", "table2"), VIEW_DEFINITION).build(); + private static final TableInfo TABLE_INFO3 = + TableInfo.builder(TableId.of("dataset", "table3"), EXTERNAL_TABLE_DEFINITION).build(); - @Rule - public ExpectedException thrown = ExpectedException.none(); + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Dataset expectedDataset; private Dataset dataset; - @Before - public void setUp() throws Exception { + private void initializeExpectedDataset(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - dataset = new Dataset(bigquery, DATASET_INFO); + expectedDataset = new Dataset(serviceMockReturnsOptions, new Dataset.BuilderImpl(DATASET_INFO)); + } + + private void initializeDataset() { + dataset = new Dataset(bigquery, new Dataset.BuilderImpl(DATASET_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(DATASET_INFO, dataset.info()); + public void testBuilder() { + initializeExpectedDataset(2); replay(bigquery); + Dataset builtDataset = new Dataset.Builder(serviceMockReturnsOptions, DATASET_ID) + .acl(ACCESS_RULES) + .creationTime(CREATION_TIME) + .defaultTableLifetime(DEFAULT_TABLE_EXPIRATION) + .description(DESCRIPTION) + .etag(ETAG) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModified(LAST_MODIFIED) + .location(LOCATION) + .selfLink(SELF_LINK) + .build(); + assertEquals(DATASET_ID, builtDataset.datasetId()); + assertEquals(ACCESS_RULES, builtDataset.acl()); + assertEquals(CREATION_TIME, builtDataset.creationTime()); + assertEquals(DEFAULT_TABLE_EXPIRATION, builtDataset.defaultTableLifetime()); + assertEquals(DESCRIPTION, builtDataset.description()); + assertEquals(ETAG, builtDataset.etag()); + assertEquals(FRIENDLY_NAME, builtDataset.friendlyName()); + assertEquals(ID, builtDataset.id()); + assertEquals(LAST_MODIFIED, builtDataset.lastModified()); + assertEquals(LOCATION, builtDataset.location()); + assertEquals(SELF_LINK, builtDataset.selfLink()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, dataset.bigquery()); + public void testToBuilder() { + initializeExpectedDataset(4); replay(bigquery); + compareDataset(expectedDataset, expectedDataset.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedDataset(1); BigQuery.DatasetOption[] expectedOptions = {BigQuery.DatasetOption.fields()}; - expect(bigquery.getDataset(DATASET_ID, expectedOptions)).andReturn(DATASET_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId(), expectedOptions)) + .andReturn(expectedDataset); replay(bigquery); + initializeDataset(); assertTrue(dataset.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedDataset(1); BigQuery.DatasetOption[] expectedOptions = {BigQuery.DatasetOption.fields()}; - expect(bigquery.getDataset(DATASET_ID, expectedOptions)).andReturn(null); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeDataset(); assertFalse(dataset.exists()); } @Test public void testReload() throws Exception { + initializeExpectedDataset(4); DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.getDataset(DATASET_ID.dataset())).andReturn(updatedInfo); + Dataset expectedDataset = + new Dataset(serviceMockReturnsOptions, new DatasetInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(expectedDataset); replay(bigquery); + initializeDataset(); Dataset updatedDataset = dataset.reload(); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + compareDataset(expectedDataset, updatedDataset); } @Test public void testReloadNull() throws Exception { - expect(bigquery.getDataset(DATASET_ID.dataset())).andReturn(null); + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(null); replay(bigquery); + initializeDataset(); assertNull(dataset.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedDataset(4); DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.getDataset(DATASET_ID.dataset(), BigQuery.DatasetOption.fields())) - .andReturn(updatedInfo); + Dataset expectedDataset = + new Dataset(serviceMockReturnsOptions, new DatasetInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset(), BigQuery.DatasetOption.fields())) + .andReturn(expectedDataset); replay(bigquery); + initializeDataset(); Dataset updatedDataset = dataset.reload(BigQuery.DatasetOption.fields()); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + compareDataset(expectedDataset, updatedDataset); } @Test - public void testUpdate() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo)).andReturn(updatedInfo); + public void testUpdate() { + initializeExpectedDataset(4); + Dataset expectedUpdatedDataset = expectedDataset.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedDataset))).andReturn(expectedUpdatedDataset); replay(bigquery); - Dataset updatedDataset = dataset.update(updatedInfo); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + initializeDataset(); + Dataset actualUpdatedDataset = dataset.update(); + compareDataset(expectedUpdatedDataset, actualUpdatedDataset); } @Test - public void testUpdateWithDifferentId() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder() - .datasetId(DatasetId.of("dataset2")) - .description("Description") - .build(); + public void testUpdateWithOptions() { + initializeExpectedDataset(4); + Dataset expectedUpdatedDataset = expectedDataset.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedDataset), eq(BigQuery.DatasetOption.fields()))) + .andReturn(expectedUpdatedDataset); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - dataset.update(updatedInfo); + initializeDataset(); + Dataset actualUpdatedDataset = dataset.update(BigQuery.DatasetOption.fields()); + compareDataset(expectedUpdatedDataset, actualUpdatedDataset); } @Test - public void testUpdateWithOptions() throws Exception { - DatasetInfo updatedInfo = DATASET_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo, BigQuery.DatasetOption.fields())).andReturn(updatedInfo); + public void testDeleteTrue() { + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(true); replay(bigquery); - Dataset updatedDataset = dataset.update(updatedInfo, BigQuery.DatasetOption.fields()); - assertSame(bigquery, updatedDataset.bigquery()); - assertEquals(updatedInfo, updatedDataset.info()); + initializeDataset(); + assertTrue(dataset.delete()); } @Test - public void testDelete() throws Exception { - expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(true); + public void testDeleteFalse() { + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(DATASET_INFO.datasetId())).andReturn(false); replay(bigquery); - assertTrue(dataset.delete()); + initializeDataset(); + assertFalse(dataset.delete()); } @Test public void testList() throws Exception { - BigQueryOptions bigqueryOptions = createStrictMock(BigQueryOptions.class); - PageImpl tableInfoPage = new PageImpl<>(null, "c", TABLE_INFO_RESULTS); - expect(bigquery.listTables(DATASET_INFO.datasetId())).andReturn(tableInfoPage); - expect(bigquery.options()).andReturn(bigqueryOptions); - expect(bigqueryOptions.service()).andReturn(bigquery); - replay(bigquery, bigqueryOptions); + initializeExpectedDataset(4); + List
tableResults = ImmutableList.of( + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO1)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO2)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO3))); + PageImpl
expectedPage = new PageImpl<>(null, "c", tableResults); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.listTables(DATASET_INFO.datasetId())).andReturn(expectedPage); + replay(bigquery); + initializeDataset(); Page
tablePage = dataset.list(); - Iterator tableInfoIterator = tableInfoPage.values().iterator(); - Iterator
tableIterator = tablePage.values().iterator(); - while (tableInfoIterator.hasNext() && tableIterator.hasNext()) { - assertEquals(tableInfoIterator.next(), tableIterator.next().info()); - } - assertFalse(tableInfoIterator.hasNext()); - assertFalse(tableIterator.hasNext()); - assertEquals(tableInfoPage.nextPageCursor(), tablePage.nextPageCursor()); - verify(bigqueryOptions); + assertArrayEquals(tableResults.toArray(), Iterables.toArray(tablePage.values(), Table.class)); + assertEquals(expectedPage.nextPageCursor(), tablePage.nextPageCursor()); } @Test public void testListWithOptions() throws Exception { - BigQueryOptions bigqueryOptions = createStrictMock(BigQueryOptions.class); - PageImpl tableInfoPage = new PageImpl<>(null, "c", TABLE_INFO_RESULTS); + initializeExpectedDataset(4); + List
tableResults = ImmutableList.of( + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO1)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO2)), + new Table(serviceMockReturnsOptions, new Table.BuilderImpl(TABLE_INFO3))); + PageImpl
expectedPage = new PageImpl<>(null, "c", tableResults); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.listTables(DATASET_INFO.datasetId(), BigQuery.TableListOption.maxResults(10L))) - .andReturn(tableInfoPage); - expect(bigquery.options()).andReturn(bigqueryOptions); - expect(bigqueryOptions.service()).andReturn(bigquery); - replay(bigquery, bigqueryOptions); + .andReturn(expectedPage); + replay(bigquery); + initializeDataset(); Page
tablePage = dataset.list(BigQuery.TableListOption.maxResults(10L)); - Iterator tableInfoIterator = tableInfoPage.values().iterator(); - Iterator
tableIterator = tablePage.values().iterator(); - while (tableInfoIterator.hasNext() && tableIterator.hasNext()) { - assertEquals(tableInfoIterator.next(), tableIterator.next().info()); - } - assertFalse(tableInfoIterator.hasNext()); - assertFalse(tableIterator.hasNext()); - assertEquals(tableInfoPage.nextPageCursor(), tablePage.nextPageCursor()); - verify(bigqueryOptions); + assertArrayEquals(tableResults.toArray(), Iterables.toArray(tablePage.values(), Table.class)); + assertEquals(expectedPage.nextPageCursor(), tablePage.nextPageCursor()); } @Test public void testGet() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); - expect(bigquery.getTable(TableId.of("dataset", "table1"))).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId())).andReturn(expectedTable); replay(bigquery); - Table table = dataset.get("table1"); + initializeDataset(); + Table table = dataset.get(TABLE_INFO1.tableId().table()); assertNotNull(table); - assertEquals(info, table.info()); + assertEquals(expectedTable, table); } @Test public void testGetNull() throws Exception { - expect(bigquery.getTable(TableId.of("dataset", "table1"))).andReturn(null); + initializeExpectedDataset(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId())).andReturn(null); replay(bigquery); - assertNull(dataset.get("table1")); + initializeDataset(); + assertNull(dataset.get(TABLE_INFO1.tableId().table())); } @Test public void testGetWithOptions() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); - expect(bigquery.getTable(TableId.of("dataset", "table1"), BigQuery.TableOption.fields())) - .andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO1.tableId(), BigQuery.TableOption.fields())) + .andReturn(expectedTable); replay(bigquery); - Table table = dataset.get("table1", BigQuery.TableOption.fields()); + initializeDataset(); + Table table = dataset.get(TABLE_INFO1.tableId().table(), BigQuery.TableOption.fields()); assertNotNull(table); - assertEquals(info, table.info()); + assertEquals(expectedTable, table); } @Test public void testCreateTable() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); - expect(bigquery.create(info)).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.create(TABLE_INFO1)).andReturn(expectedTable); replay(bigquery); - Table table = dataset.create("table1", TABLE_DEFINITION); - assertEquals(info, table.info()); + initializeDataset(); + Table table = dataset.create(TABLE_INFO1.tableId().table(), TABLE_DEFINITION); + assertEquals(expectedTable, table); } @Test public void testCreateTableWithOptions() throws Exception { - TableInfo info = TableInfo.builder(TableId.of("dataset", "table1"), TABLE_DEFINITION).build(); - expect(bigquery.create(info, BigQuery.TableOption.fields())).andReturn(info); + initializeExpectedDataset(2); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO1)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.create(TABLE_INFO1, BigQuery.TableOption.fields())).andReturn(expectedTable); replay(bigquery); - Table table = dataset.create("table1", TABLE_DEFINITION, BigQuery.TableOption.fields()); - assertEquals(info, table.info()); + initializeDataset(); + Table table = dataset.create(TABLE_INFO1.tableId().table(), TABLE_DEFINITION, + BigQuery.TableOption.fields()); + assertEquals(expectedTable, table); } @Test - public void testStaticGet() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(DATASET_INFO); + public void testBigquery() { + initializeExpectedDataset(1); replay(bigquery); - Dataset loadedDataset = Dataset.get(bigquery, DATASET_INFO.datasetId().dataset()); - assertNotNull(loadedDataset); - assertEquals(DATASET_INFO, loadedDataset.info()); + assertSame(serviceMockReturnsOptions, expectedDataset.bigquery()); } @Test - public void testStaticGetNull() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset())).andReturn(null); + public void testToAndFromPb() { + initializeExpectedDataset(4); replay(bigquery); - assertNull(Dataset.get(bigquery, DATASET_INFO.datasetId().dataset())); + compareDataset(expectedDataset, + Dataset.fromPb(serviceMockReturnsOptions, expectedDataset.toPb())); } - @Test - public void testStaticGetWithOptions() throws Exception { - expect(bigquery.getDataset(DATASET_INFO.datasetId().dataset(), BigQuery.DatasetOption.fields())) - .andReturn(DATASET_INFO); - replay(bigquery); - Dataset loadedDataset = Dataset.get(bigquery, DATASET_INFO.datasetId().dataset(), - BigQuery.DatasetOption.fields()); - assertNotNull(loadedDataset); - assertEquals(DATASET_INFO, loadedDataset.info()); + private void compareDataset(Dataset expected, Dataset value) { + assertEquals(expected, value); + compareDatasetInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); + } + + private void compareDatasetInfo(DatasetInfo expected, DatasetInfo value) { + assertEquals(expected, value); + assertEquals(expected.datasetId(), value.datasetId()); + assertEquals(expected.description(), value.description()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.friendlyName(), value.friendlyName()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.location(), value.location()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.acl(), value.acl()); + assertEquals(expected.creationTime(), value.creationTime()); + assertEquals(expected.defaultTableLifetime(), value.defaultTableLifetime()); + assertEquals(expected.lastModified(), value.lastModified()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java index 90b602d978e0..db51706fff5a 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/JobTest.java @@ -16,161 +16,243 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import org.junit.After; -import org.junit.Before; import org.junit.Test; public class JobTest { - private static final JobId JOB_ID = JobId.of("dataset", "job"); + private static final JobId JOB_ID = JobId.of("project", "job"); private static final TableId TABLE_ID1 = TableId.of("dataset", "table1"); private static final TableId TABLE_ID2 = TableId.of("dataset", "table2"); - private static final JobInfo JOB_INFO = - JobInfo.of(JOB_ID, CopyJobConfiguration.of(TABLE_ID1, TABLE_ID2)); + private static final String ETAG = "etag"; + private static final String ID = "id"; + private static final String SELF_LINK = "selfLink"; + private static final String EMAIL = "email"; + private static final JobStatus JOB_STATUS = new JobStatus(JobStatus.State.DONE); + private static final JobStatistics COPY_JOB_STATISTICS = JobStatistics.builder() + .creationTime(1L) + .endTime(3L) + .startTime(2L) + .build(); + private static final CopyJobConfiguration COPY_CONFIGURATION = + CopyJobConfiguration.of(TABLE_ID1, TABLE_ID2); + private static final JobInfo JOB_INFO = JobInfo.builder(COPY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(COPY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Job expectedJob; private Job job; - @Before - public void setUp() throws Exception { + private void initializeExpectedJob(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - job = new Job(bigquery, JOB_INFO); + expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(JOB_INFO)); + } + + private void initializeJob() { + job = new Job(bigquery, new JobInfo.BuilderImpl(JOB_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(JOB_INFO, job.info()); + public void testBuilder() { + initializeExpectedJob(2); replay(bigquery); + Job builtJob = new Job.Builder(serviceMockReturnsOptions, COPY_CONFIGURATION) + .jobId(JOB_ID) + .statistics(COPY_JOB_STATISTICS) + .jobId(JOB_ID) + .etag(ETAG) + .id(ID) + .selfLink(SELF_LINK) + .userEmail(EMAIL) + .status(JOB_STATUS) + .build(); + assertEquals(ETAG, builtJob.etag()); + assertEquals(ID, builtJob.id()); + assertEquals(SELF_LINK, builtJob.selfLink()); + assertEquals(EMAIL, builtJob.userEmail()); + assertEquals(JOB_ID, builtJob.jobId()); + assertEquals(JOB_STATUS, builtJob.status()); + assertEquals(COPY_CONFIGURATION, builtJob.configuration()); + assertEquals(COPY_JOB_STATISTICS, builtJob.statistics()); + assertSame(serviceMockReturnsOptions, builtJob.bigquery()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, job.bigquery()); + public void testToBuilder() { + initializeExpectedJob(4); replay(bigquery); + compareJob(expectedJob, expectedJob.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields()}; - expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(JOB_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(expectedJob); replay(bigquery); + initializeJob(); assertTrue(job.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields()}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeJob(); assertFalse(job.exists()); } @Test public void testIsDone_True() throws Exception { + initializeExpectedJob(2); + BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; JobStatus status = createStrictMock(JobStatus.class); expect(status.state()).andReturn(JobStatus.State.DONE); - BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)) - .andReturn(JOB_INFO.toBuilder().status(status).build()); + .andReturn(expectedJob.toBuilder().status(status).build()); replay(status, bigquery); + initializeJob(); assertTrue(job.isDone()); verify(status); } @Test public void testIsDone_False() throws Exception { + initializeExpectedJob(2); + BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; JobStatus status = createStrictMock(JobStatus.class); expect(status.state()).andReturn(JobStatus.State.RUNNING); - BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)) - .andReturn(JOB_INFO.toBuilder().status(status).build()); + .andReturn(expectedJob.toBuilder().status(status).build()); replay(status, bigquery); + initializeJob(); assertFalse(job.isDone()); verify(status); } @Test public void testIsDone_NotExists() throws Exception { + initializeExpectedJob(1); BigQuery.JobOption[] expectedOptions = {BigQuery.JobOption.fields(BigQuery.JobField.STATUS)}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeJob(); assertFalse(job.isDone()); } @Test public void testReload() throws Exception { + initializeExpectedJob(4); JobInfo updatedInfo = JOB_INFO.toBuilder().etag("etag").build(); - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(updatedInfo); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(expectedJob); replay(bigquery); + initializeJob(); Job updatedJob = job.reload(); - assertSame(bigquery, updatedJob.bigquery()); - assertEquals(updatedInfo, updatedJob.info()); + compareJob(expectedJob, updatedJob); } @Test public void testReloadNull() throws Exception { + initializeExpectedJob(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(null); replay(bigquery); + initializeJob(); assertNull(job.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedJob(4); JobInfo updatedInfo = JOB_INFO.toBuilder().etag("etag").build(); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getJob(JOB_INFO.jobId().job(), BigQuery.JobOption.fields())) - .andReturn(updatedInfo); + .andReturn(expectedJob); replay(bigquery); + initializeJob(); Job updatedJob = job.reload(BigQuery.JobOption.fields()); - assertSame(bigquery, updatedJob.bigquery()); - assertEquals(updatedInfo, updatedJob.info()); + compareJob(expectedJob, updatedJob); } @Test public void testCancel() throws Exception { + initializeExpectedJob(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.cancel(JOB_INFO.jobId())).andReturn(true); replay(bigquery); + initializeJob(); assertTrue(job.cancel()); } @Test - public void testGet() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(JOB_INFO); + public void testBigquery() { + initializeExpectedJob(1); replay(bigquery); - Job loadedJob = Job.get(bigquery, JOB_INFO.jobId().job()); - assertNotNull(loadedJob); - assertEquals(JOB_INFO, loadedJob.info()); + assertSame(serviceMockReturnsOptions, expectedJob.bigquery()); } @Test - public void testGetNull() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job())).andReturn(null); + public void testToAndFromPb() { + initializeExpectedJob(4); replay(bigquery); - assertNull(Job.get(bigquery, JOB_INFO.jobId().job())); + compareJob(expectedJob, Job.fromPb(serviceMockReturnsOptions, expectedJob.toPb())); } - @Test - public void testGetWithOptions() throws Exception { - expect(bigquery.getJob(JOB_INFO.jobId().job(), BigQuery.JobOption.fields())) - .andReturn(JOB_INFO); - replay(bigquery); - Job loadedJob = Job.get(bigquery, JOB_INFO.jobId().job(), BigQuery.JobOption.fields()); - assertNotNull(loadedJob); - assertEquals(JOB_INFO, loadedJob.info()); + private void compareJob(Job expected, Job value) { + assertEquals(expected, value); + compareJobInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); + } + + private void compareJobInfo(JobInfo expected, JobInfo value) { + assertEquals(expected, value); + assertEquals(expected.hashCode(), value.hashCode()); + assertEquals(expected.toString(), value.toString()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.jobId(), value.jobId()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.status(), value.status()); + assertEquals(expected.statistics(), value.statistics()); + assertEquals(expected.userEmail(), value.userEmail()); + assertEquals(expected.configuration(), value.configuration()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java index 62a88c1860cd..267ae161b7aa 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/RemoteBigQueryHelperTest.java @@ -65,6 +65,7 @@ public class RemoteBigQueryHelperTest { @Rule public ExpectedException thrown = ExpectedException.none(); + @Test public void testForceDelete() throws InterruptedException, ExecutionException { BigQuery bigqueryMock = EasyMock.createMock(BigQuery.class); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java index 0c54ea627a67..d877bff2138c 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/SerializationTest.java @@ -224,6 +224,12 @@ public class SerializationTest { .jobCompleted(true) .result(QUERY_RESULT) .build(); + private static final BigQuery BIGQUERY = + BigQueryOptions.builder().projectId("p1").build().service(); + private static final Dataset DATASET = + new Dataset(BIGQUERY, new DatasetInfo.BuilderImpl(DATASET_INFO)); + private static final Table TABLE = new Table(BIGQUERY, new TableInfo.BuilderImpl(TABLE_INFO)); + private static final Job JOB = new Job(BIGQUERY, new JobInfo.BuilderImpl(JOB_INFO)); @Test public void testServiceOptions() throws Exception { @@ -256,7 +262,7 @@ public void testModelAndRequests() throws Exception { BigQuery.DatasetOption.fields(), BigQuery.DatasetDeleteOption.deleteContents(), BigQuery.DatasetListOption.all(), BigQuery.TableOption.fields(), BigQuery.TableListOption.maxResults(42L), BigQuery.JobOption.fields(), - BigQuery.JobListOption.allUsers()}; + BigQuery.JobListOption.allUsers(), DATASET, TABLE, JOB}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java index 3b16593a7f79..4866ee9ab8ec 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/TableTest.java @@ -16,13 +16,14 @@ package com.google.gcloud.bigquery; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; @@ -35,16 +36,21 @@ import com.google.gcloud.bigquery.InsertAllRequest.RowToInsert; import org.junit.After; -import org.junit.Before; -import org.junit.Rule; import org.junit.Test; -import org.junit.rules.ExpectedException; import java.util.Iterator; import java.util.List; public class TableTest { + private static final String ETAG = "etag"; + private static final String ID = "project:dataset:table1"; + private static final String SELF_LINK = "selfLink"; + private static final String FRIENDLY_NAME = "friendlyName"; + private static final String DESCRIPTION = "description"; + private static final Long CREATION_TIME = 10L; + private static final Long EXPIRATION_TIME = 100L; + private static final Long LAST_MODIFIED_TIME = 20L; private static final TableId TABLE_ID1 = TableId.of("dataset", "table1"); private static final TableId TABLE_ID2 = TableId.of("dataset", "table2"); private static final CopyJobConfiguration COPY_JOB_CONFIGURATION = @@ -77,148 +83,198 @@ public class TableTest { private static final Iterable> ROWS = ImmutableList.of( (List) ImmutableList.of(FIELD_VALUE1), ImmutableList.of(FIELD_VALUE2)); - @Rule - public ExpectedException thrown = ExpectedException.none(); + private BigQuery serviceMockReturnsOptions = createStrictMock(BigQuery.class); + private BigQueryOptions mockOptions = createMock(BigQueryOptions.class); private BigQuery bigquery; + private Table expectedTable; private Table table; - @Before - public void setUp() throws Exception { + private void initializeExpectedTable(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); bigquery = createStrictMock(BigQuery.class); - table = new Table(bigquery, TABLE_INFO); + expectedTable = new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(TABLE_INFO)); + } + + private void initializeTable() { + table = new Table(bigquery, new TableInfo.BuilderImpl(TABLE_INFO)); } @After public void tearDown() throws Exception { - verify(bigquery); + verify(bigquery, serviceMockReturnsOptions); } @Test - public void testInfo() throws Exception { - assertEquals(TABLE_INFO, table.info()); + public void testBuilder() { + initializeExpectedTable(2); replay(bigquery); + Table builtTable = new Table.Builder(serviceMockReturnsOptions, TABLE_ID1, TABLE_DEFINITION) + .creationTime(CREATION_TIME) + .description(DESCRIPTION) + .etag(ETAG) + .expirationTime(EXPIRATION_TIME) + .friendlyName(FRIENDLY_NAME) + .id(ID) + .lastModifiedTime(LAST_MODIFIED_TIME) + .selfLink(SELF_LINK) + .build(); + assertEquals(TABLE_ID1, builtTable.tableId()); + assertEquals(CREATION_TIME, builtTable.creationTime()); + assertEquals(DESCRIPTION, builtTable.description()); + assertEquals(ETAG, builtTable.etag()); + assertEquals(EXPIRATION_TIME, builtTable.expirationTime()); + assertEquals(FRIENDLY_NAME, builtTable.friendlyName()); + assertEquals(ID, builtTable.id()); + assertEquals(LAST_MODIFIED_TIME, builtTable.lastModifiedTime()); + assertEquals(TABLE_DEFINITION, builtTable.definition()); + assertEquals(SELF_LINK, builtTable.selfLink()); + assertSame(serviceMockReturnsOptions, builtTable.bigquery()); } @Test - public void testBigQuery() throws Exception { - assertSame(bigquery, table.bigquery()); + public void testToBuilder() { + initializeExpectedTable(4); replay(bigquery); + compareTable(expectedTable, expectedTable.toBuilder().build()); } @Test public void testExists_True() throws Exception { + initializeExpectedTable(1); BigQuery.TableOption[] expectedOptions = {BigQuery.TableOption.fields()}; - expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(TABLE_INFO); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(expectedTable); replay(bigquery); + initializeTable(); assertTrue(table.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedTable(1); BigQuery.TableOption[] expectedOptions = {BigQuery.TableOption.fields()}; + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId(), expectedOptions)).andReturn(null); replay(bigquery); + initializeTable(); assertFalse(table.exists()); } @Test public void testReload() throws Exception { + initializeExpectedTable(4); TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(updatedInfo); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(expectedTable); replay(bigquery); + initializeTable(); Table updatedTable = table.reload(); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + compareTable(expectedTable, updatedTable); } @Test public void testReloadNull() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); replay(bigquery); + initializeTable(); assertNull(table.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedTable(4); TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); + Table expectedTable = + new Table(serviceMockReturnsOptions, new TableInfo.BuilderImpl(updatedInfo)); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(updatedInfo); + .andReturn(expectedTable); replay(bigquery); + initializeTable(); Table updatedTable = table.reload(BigQuery.TableOption.fields()); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); - } - - @Test - public void testUpdate() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo)).andReturn(updatedInfo); - replay(bigquery); - Table updatedTable = table.update(updatedInfo); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + compareTable(expectedTable, updatedTable); } @Test - public void testUpdateWithDifferentId() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder() - .tableId(TableId.of("dataset", "table3")) - .description("Description") - .build(); + public void testUpdate() { + initializeExpectedTable(4); + Table expectedUpdatedTable = expectedTable.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedTable))).andReturn(expectedUpdatedTable); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - table.update(updatedInfo); + initializeTable(); + Table actualUpdatedTable = table.update(); + compareTable(expectedUpdatedTable, actualUpdatedTable); } @Test - public void testUpdateWithDifferentDatasetId() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder() - .tableId(TableId.of("dataset1", "table1")) - .description("Description") - .build(); + public void testUpdateWithOptions() { + initializeExpectedTable(4); + Table expectedUpdatedTable = expectedTable.toBuilder().description("Description").build(); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.update(eq(expectedTable), eq(BigQuery.TableOption.fields()))) + .andReturn(expectedUpdatedTable); replay(bigquery); - thrown.expect(IllegalArgumentException.class); - table.update(updatedInfo); + initializeTable(); + Table actualUpdatedTable = table.update(BigQuery.TableOption.fields()); + compareTable(expectedUpdatedTable, actualUpdatedTable); } @Test - public void testUpdateWithOptions() throws Exception { - TableInfo updatedInfo = TABLE_INFO.toBuilder().description("Description").build(); - expect(bigquery.update(updatedInfo, BigQuery.TableOption.fields())).andReturn(updatedInfo); + public void testDeleteTrue() { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(true); replay(bigquery); - Table updatedTable = table.update(updatedInfo, BigQuery.TableOption.fields()); - assertSame(bigquery, updatedTable.bigquery()); - assertEquals(updatedInfo, updatedTable.info()); + initializeTable(); + assertTrue(table.delete()); } @Test - public void testDelete() throws Exception { - expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(true); + public void testDeleteFalse() { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); + expect(bigquery.delete(TABLE_INFO.tableId())).andReturn(false); replay(bigquery); - assertTrue(table.delete()); + initializeTable(); + assertFalse(table.delete()); } @Test public void testInsert() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.insertAll(INSERT_ALL_REQUEST)).andReturn(EMPTY_INSERT_ALL_RESPONSE); replay(bigquery); + initializeTable(); InsertAllResponse response = table.insert(ROWS_TO_INSERT); assertSame(EMPTY_INSERT_ALL_RESPONSE, response); } @Test public void testInsertComplete() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); expect(bigquery.insertAll(INSERT_ALL_REQUEST_COMPLETE)).andReturn(EMPTY_INSERT_ALL_RESPONSE); replay(bigquery); + initializeTable(); InsertAllResponse response = table.insert(ROWS_TO_INSERT, true, true); assertSame(EMPTY_INSERT_ALL_RESPONSE, response); } @Test public void testList() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); PageImpl> tableDataPage = new PageImpl<>(null, "c", ROWS); expect(bigquery.listTableData(TABLE_ID1)).andReturn(tableDataPage); replay(bigquery); + initializeTable(); Page> dataPage = table.list(); Iterator> tableDataIterator = tableDataPage.values().iterator(); Iterator> dataIterator = dataPage.values().iterator(); @@ -227,10 +283,13 @@ public void testList() throws Exception { @Test public void testListWithOptions() throws Exception { + initializeExpectedTable(1); + expect(bigquery.options()).andReturn(mockOptions); PageImpl> tableDataPage = new PageImpl<>(null, "c", ROWS); expect(bigquery.listTableData(TABLE_ID1, BigQuery.TableDataListOption.maxResults(10L))) .andReturn(tableDataPage); replay(bigquery); + initializeTable(); Page> dataPage = table.list(BigQuery.TableDataListOption.maxResults(10L)); Iterator> tableDataIterator = tableDataPage.values().iterator(); Iterator> dataIterator = dataPage.values().iterator(); @@ -239,108 +298,110 @@ public void testListWithOptions() throws Exception { @Test public void testCopyFromString() throws Exception { - expect(bigquery.create(COPY_JOB_INFO)).andReturn(COPY_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(COPY_JOB_INFO)); + expect(bigquery.create(COPY_JOB_INFO)) + .andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.copy(TABLE_ID2.dataset(), TABLE_ID2.table()); - assertSame(bigquery, job.bigquery()); - assertEquals(COPY_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testCopyFromId() throws Exception { - expect(bigquery.create(COPY_JOB_INFO)).andReturn(COPY_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(COPY_JOB_INFO)); + expect(bigquery.create(COPY_JOB_INFO)).andReturn(expectedJob); replay(bigquery); - Job job = table.copy(TABLE_ID2); - assertSame(bigquery, job.bigquery()); - assertEquals(COPY_JOB_INFO, job.info()); + initializeTable(); + Job job = table.copy(TABLE_ID2.dataset(), TABLE_ID2.table()); + assertSame(expectedJob, job); } @Test public void testLoadDataUri() throws Exception { - expect(bigquery.create(LOAD_JOB_INFO)).andReturn(LOAD_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(LOAD_JOB_INFO)); + expect(bigquery.create(LOAD_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.load(FormatOptions.json(), "URI"); - assertSame(bigquery, job.bigquery()); - assertEquals(LOAD_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testLoadDataUris() throws Exception { - expect(bigquery.create(LOAD_JOB_INFO)).andReturn(LOAD_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(LOAD_JOB_INFO)); + expect(bigquery.create(LOAD_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.load(FormatOptions.json(), ImmutableList.of("URI")); - assertSame(bigquery, job.bigquery()); - assertEquals(LOAD_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testExtractDataUri() throws Exception { - expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(EXTRACT_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(EXTRACT_JOB_INFO)); + expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.extract("CSV", "URI"); - assertSame(bigquery, job.bigquery()); - assertEquals(EXTRACT_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test public void testExtractDataUris() throws Exception { - expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(EXTRACT_JOB_INFO); + initializeExpectedTable(2); + expect(bigquery.options()).andReturn(mockOptions); + Job expectedJob = new Job(serviceMockReturnsOptions, new JobInfo.BuilderImpl(EXTRACT_JOB_INFO)); + expect(bigquery.create(EXTRACT_JOB_INFO)).andReturn(expectedJob); replay(bigquery); + initializeTable(); Job job = table.extract("CSV", ImmutableList.of("URI")); - assertSame(bigquery, job.bigquery()); - assertEquals(EXTRACT_JOB_INFO, job.info()); + assertSame(expectedJob, job); } @Test - public void testGetFromId() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(TABLE_INFO); + public void testBigquery() { + initializeExpectedTable(1); replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_INFO.tableId()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + assertSame(serviceMockReturnsOptions, expectedTable.bigquery()); } @Test - public void testGetFromStrings() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(TABLE_INFO); + public void testToAndFromPb() { + initializeExpectedTable(4); replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + compareTable(expectedTable, Table.fromPb(serviceMockReturnsOptions, expectedTable.toPb())); } - @Test - public void testGetFromIdNull() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); - replay(bigquery); - assertNull(Table.get(bigquery, TABLE_INFO.tableId())); - } - - @Test - public void testGetFromStringsNull() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId())).andReturn(null); - replay(bigquery); - assertNull(Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table())); + private void compareTable(Table expected, Table value) { + assertEquals(expected, value); + compareTableInfo(expected, value); + assertEquals(expected.bigquery().options(), value.bigquery().options()); } - @Test - public void testGetFromIdWithOptions() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(TABLE_INFO); - replay(bigquery); - Table loadedTable = Table.get(bigquery, TABLE_INFO.tableId(), BigQuery.TableOption.fields()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); - } - - @Test - public void testGetFromStringsWithOptions() throws Exception { - expect(bigquery.getTable(TABLE_INFO.tableId(), BigQuery.TableOption.fields())) - .andReturn(TABLE_INFO); - replay(bigquery); - Table loadedTable = - Table.get(bigquery, TABLE_ID1.dataset(), TABLE_ID1.table(), BigQuery.TableOption.fields()); - assertNotNull(loadedTable); - assertEquals(TABLE_INFO, loadedTable.info()); + private void compareTableInfo(TableInfo expected, TableInfo value) { + assertEquals(expected, value); + assertEquals(expected.tableId(), value.tableId()); + assertEquals(expected.definition(), value.definition()); + assertEquals(expected.creationTime(), value.creationTime()); + assertEquals(expected.description(), value.description()); + assertEquals(expected.etag(), value.etag()); + assertEquals(expected.expirationTime(), value.expirationTime()); + assertEquals(expected.friendlyName(), value.friendlyName()); + assertEquals(expected.id(), value.id()); + assertEquals(expected.lastModifiedTime(), value.lastModifiedTime()); + assertEquals(expected.selfLink(), value.selfLink()); + assertEquals(expected.definition(), value.definition()); + assertEquals(expected.hashCode(), value.hashCode()); } } diff --git a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java similarity index 77% rename from gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java rename to gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java index 0928c04ea6d2..63a0551ece33 100644 --- a/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java +++ b/gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/it/ITBigQueryTest.java @@ -14,14 +14,8 @@ * limitations under the License. */ -package com.google.gcloud.bigquery; - -import static com.google.gcloud.bigquery.BigQuery.DatasetField; -import static com.google.gcloud.bigquery.BigQuery.JobField; -import static com.google.gcloud.bigquery.BigQuery.JobListOption; -import static com.google.gcloud.bigquery.BigQuery.JobOption; -import static com.google.gcloud.bigquery.BigQuery.TableField; -import static com.google.gcloud.bigquery.BigQuery.TableOption; +package com.google.gcloud.bigquery.it; + import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; @@ -32,7 +26,43 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.gcloud.Page; +import com.google.gcloud.WriteChannel; +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQuery.DatasetField; import com.google.gcloud.bigquery.BigQuery.DatasetOption; +import com.google.gcloud.bigquery.BigQuery.JobField; +import com.google.gcloud.bigquery.BigQuery.JobListOption; +import com.google.gcloud.bigquery.BigQuery.JobOption; +import com.google.gcloud.bigquery.BigQuery.TableField; +import com.google.gcloud.bigquery.BigQuery.TableOption; +import com.google.gcloud.bigquery.BigQueryError; +import com.google.gcloud.bigquery.BigQueryException; +import com.google.gcloud.bigquery.CopyJobConfiguration; +import com.google.gcloud.bigquery.Dataset; +import com.google.gcloud.bigquery.DatasetId; +import com.google.gcloud.bigquery.DatasetInfo; +import com.google.gcloud.bigquery.ExternalTableDefinition; +import com.google.gcloud.bigquery.ExtractJobConfiguration; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FieldValue; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.InsertAllRequest; +import com.google.gcloud.bigquery.InsertAllResponse; +import com.google.gcloud.bigquery.Job; +import com.google.gcloud.bigquery.JobInfo; +import com.google.gcloud.bigquery.JobStatistics; +import com.google.gcloud.bigquery.LoadJobConfiguration; +import com.google.gcloud.bigquery.QueryJobConfiguration; +import com.google.gcloud.bigquery.QueryRequest; +import com.google.gcloud.bigquery.QueryResponse; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; +import com.google.gcloud.bigquery.TableDefinition; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; +import com.google.gcloud.bigquery.ViewDefinition; +import com.google.gcloud.bigquery.WriteChannelConfiguration; import com.google.gcloud.bigquery.testing.RemoteBigQueryHelper; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.BucketInfo; @@ -45,7 +75,6 @@ import org.junit.Test; import org.junit.rules.Timeout; -import java.io.FileNotFoundException; import java.io.IOException; import java.nio.ByteBuffer; import java.nio.charset.StandardCharsets; @@ -58,7 +87,7 @@ public class ITBigQueryTest { - private static final Logger log = Logger.getLogger(ITBigQueryTest.class.getName()); + private static final Logger LOG = Logger.getLogger(ITBigQueryTest.class.getName()); private static final String DATASET = RemoteBigQueryHelper.generateDatasetName(); private static final String DESCRIPTION = "Test dataset"; private static final String OTHER_DATASET = RemoteBigQueryHelper.generateDatasetName(); @@ -140,7 +169,7 @@ public class ITBigQueryTest { public Timeout globalTimeout = Timeout.seconds(300); @BeforeClass - public static void beforeClass() throws IOException, InterruptedException { + public static void beforeClass() throws InterruptedException { RemoteBigQueryHelper bigqueryHelper = RemoteBigQueryHelper.create(); RemoteGcsHelper gcsHelper = RemoteGcsHelper.create(); bigquery = bigqueryHelper.options().service(); @@ -157,10 +186,9 @@ public static void beforeClass() throws IOException, InterruptedException { .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) .schema(TABLE_SCHEMA) .build(); - JobInfo job = bigquery.create(JobInfo.of(configuration)); - while (job.status().state() != JobStatus.State.DONE) { + Job job = bigquery.create(JobInfo.of(configuration)); + while (!job.isDone()) { Thread.sleep(1000); - job = bigquery.getJob(job.jobId()); } assertNull(job.status().error()); } @@ -170,16 +198,17 @@ public static void afterClass() throws ExecutionException, InterruptedException if (bigquery != null) { RemoteBigQueryHelper.forceDelete(bigquery, DATASET); } - if (storage != null && !RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS)) { - if (log.isLoggable(Level.WARNING)) { - log.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); + if (storage != null) { + boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 10, TimeUnit.SECONDS); + if (!wasDeleted && LOG.isLoggable(Level.WARNING)) { + LOG.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); } } } @Test public void testGetDataset() { - DatasetInfo dataset = bigquery.getDataset(DATASET); + Dataset dataset = bigquery.getDataset(DATASET); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(DATASET, dataset.datasetId().dataset()); assertEquals(DESCRIPTION, dataset.description()); @@ -192,7 +221,7 @@ public void testGetDataset() { @Test public void testGetDatasetWithSelectedFields() { - DatasetInfo dataset = bigquery.getDataset(DATASET, + Dataset dataset = bigquery.getDataset(DATASET, DatasetOption.fields(DatasetField.CREATION_TIME)); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(DATASET, dataset.datasetId().dataset()); @@ -210,29 +239,29 @@ public void testGetDatasetWithSelectedFields() { @Test public void testUpdateDataset() { - DatasetInfo dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) + Dataset dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) .description("Some Description") .build()); assertNotNull(dataset); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(OTHER_DATASET, dataset.datasetId().dataset()); assertEquals("Some Description", dataset.description()); - DatasetInfo updatedDataset = + Dataset updatedDataset = bigquery.update(dataset.toBuilder().description("Updated Description").build()); assertEquals("Updated Description", updatedDataset.description()); - assertTrue(bigquery.delete(OTHER_DATASET)); + assertTrue(dataset.delete()); } @Test public void testUpdateDatasetWithSelectedFields() { - DatasetInfo dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) + Dataset dataset = bigquery.create(DatasetInfo.builder(OTHER_DATASET) .description("Some Description") .build()); assertNotNull(dataset); assertEquals(bigquery.options().projectId(), dataset.datasetId().project()); assertEquals(OTHER_DATASET, dataset.datasetId().dataset()); assertEquals("Some Description", dataset.description()); - DatasetInfo updatedDataset = + Dataset updatedDataset = bigquery.update(dataset.toBuilder().description("Updated Description").build(), DatasetOption.fields(DatasetField.DESCRIPTION)); assertEquals("Updated Description", updatedDataset.description()); @@ -245,7 +274,7 @@ public void testUpdateDatasetWithSelectedFields() { assertNull(updatedDataset.lastModified()); assertNull(updatedDataset.location()); assertNull(updatedDataset.selfLink()); - assertTrue(bigquery.delete(OTHER_DATASET)); + assertTrue(dataset.delete()); } @Test @@ -258,21 +287,21 @@ public void testCreateAndGetTable() { String tableName = "test_create_and_get_table"; TableId tableId = TableId.of(DATASET, tableName); StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); - TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, tableDefinition)); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - TableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo.definition() instanceof StandardTableDefinition); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(TableDefinition.Type.TABLE, remoteTableInfo.definition().type()); - assertEquals(TABLE_SCHEMA, remoteTableInfo.definition().schema()); - assertNotNull(remoteTableInfo.creationTime()); - assertNotNull(remoteTableInfo.lastModifiedTime()); - assertNotNull(remoteTableInfo.definition().numBytes()); - assertNotNull(remoteTableInfo.definition().numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof StandardTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TableDefinition.Type.TABLE, remoteTable.definition().type()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); + assertNotNull(remoteTable.creationTime()); + assertNotNull(remoteTable.lastModifiedTime()); + assertNotNull(remoteTable.definition().numBytes()); + assertNotNull(remoteTable.definition().numRows()); + assertTrue(remoteTable.delete()); } @Test @@ -280,22 +309,22 @@ public void testCreateAndGetTableWithSelectedField() { String tableName = "test_create_and_get_selected_fields_table"; TableId tableId = TableId.of(DATASET, tableName); StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); - TableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, tableDefinition)); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - TableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName, + Table createdTable = bigquery.create(TableInfo.of(tableId, tableDefinition)); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName, TableOption.fields(TableField.CREATION_TIME)); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo.definition() instanceof StandardTableDefinition); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(TableDefinition.Type.TABLE, remoteTableInfo.definition().type()); - assertNotNull(remoteTableInfo.creationTime()); - assertNull(remoteTableInfo.definition().schema()); - assertNull(remoteTableInfo.lastModifiedTime()); - assertNull(remoteTableInfo.definition().numBytes()); - assertNull(remoteTableInfo.definition().numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof StandardTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TableDefinition.Type.TABLE, remoteTable.definition().type()); + assertNotNull(remoteTable.creationTime()); + assertNull(remoteTable.definition().schema()); + assertNull(remoteTable.lastModifiedTime()); + assertNull(remoteTable.definition().numBytes()); + assertNull(remoteTable.definition().numRows()); + assertTrue(remoteTable.delete()); } @Test @@ -305,15 +334,15 @@ public void testCreateExternalTable() throws InterruptedException { ExternalTableDefinition externalTableDefinition = ExternalTableDefinition.of( "gs://" + BUCKET + "/" + JSON_LOAD_FILE, TABLE_SCHEMA, FormatOptions.json()); TableInfo tableInfo = TableInfo.of(tableId, externalTableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - TableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertTrue(remoteTableInfo.definition() instanceof ExternalTableDefinition); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertEquals(TABLE_SCHEMA, remoteTableInfo.definition().schema()); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertTrue(remoteTable.definition() instanceof ExternalTableDefinition); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); QueryRequest request = QueryRequest.builder( "SELECT TimestampField, StringField, IntegerField, BooleanField FROM " + DATASET + "." + tableName) @@ -345,7 +374,7 @@ public void testCreateExternalTable() throws InterruptedException { rowCount++; } assertEquals(4, rowCount); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(remoteTable.delete()); } @Test @@ -356,14 +385,14 @@ public void testCreateViewTable() throws InterruptedException { ViewDefinition.of("SELECT TimestampField, StringField, BooleanField FROM " + DATASET + "." + TABLE_ID.table()); TableInfo tableInfo = TableInfo.of(tableId, viewDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(tableName, createdTableInfo.tableId().table()); - TableInfo remoteTableInfo = bigquery.getTable(DATASET, tableName); - assertNotNull(remoteTableInfo); - assertEquals(createdTableInfo.tableId(), remoteTableInfo.tableId()); - assertTrue(remoteTableInfo.definition() instanceof ViewDefinition); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(tableName, createdTable.tableId().table()); + Table remoteTable = bigquery.getTable(DATASET, tableName); + assertNotNull(remoteTable); + assertEquals(createdTable.tableId(), remoteTable.tableId()); + assertTrue(remoteTable.definition() instanceof ViewDefinition); Schema expectedSchema = Schema.builder() .addField( Field.builder("TimestampField", Field.Type.timestamp()) @@ -378,7 +407,7 @@ public void testCreateViewTable() throws InterruptedException { .mode(Field.Mode.NULLABLE) .build()) .build(); - assertEquals(expectedSchema, remoteTableInfo.definition().schema()); + assertEquals(expectedSchema, remoteTable.definition().schema()); QueryRequest request = QueryRequest.builder("SELECT * FROM " + tableName) .defaultDataset(DatasetId.of(DATASET)) .maxWaitTime(60000L) @@ -403,7 +432,7 @@ public void testCreateViewTable() throws InterruptedException { rowCount++; } assertEquals(2, rowCount); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(remoteTable.delete()); } @Test @@ -411,18 +440,18 @@ public void testListTables() { String tableName = "test_list_tables"; StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - Page tables = bigquery.listTables(DATASET); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Page
tables = bigquery.listTables(DATASET); boolean found = false; - Iterator tableIterator = tables.values().iterator(); + Iterator
tableIterator = tables.values().iterator(); while (tableIterator.hasNext() && !found) { - if (tableIterator.next().tableId().equals(createdTableInfo.tableId())) { + if (tableIterator.next().tableId().equals(createdTable.tableId())) { found = true; } } assertTrue(found); - assertTrue(bigquery.delete(DATASET, tableName)); + assertTrue(createdTable.delete()); } @Test @@ -430,15 +459,15 @@ public void testUpdateTable() { String tableName = "test_update_table"; StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - TableInfo updatedTableInfo = bigquery.update(tableInfo.toBuilder() - .description("newDescription").build()); - assertEquals(DATASET, updatedTableInfo.tableId().dataset()); - assertEquals(tableName, updatedTableInfo.tableId().table()); - assertEquals(TABLE_SCHEMA, updatedTableInfo.definition().schema()); - assertEquals("newDescription", updatedTableInfo.description()); - assertTrue(bigquery.delete(DATASET, tableName)); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Table updatedTable = + bigquery.update(tableInfo.toBuilder().description("newDescription").build()); + assertEquals(DATASET, updatedTable.tableId().dataset()); + assertEquals(tableName, updatedTable.tableId().table()); + assertEquals(TABLE_SCHEMA, updatedTable.definition().schema()); + assertEquals("newDescription", updatedTable.description()); + assertTrue(updatedTable.delete()); } @Test @@ -446,19 +475,19 @@ public void testUpdateTableWithSelectedFields() { String tableName = "test_update_with_selected_fields_table"; StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(TableId.of(DATASET, tableName), tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - TableInfo updatedTableInfo = bigquery.update(tableInfo.toBuilder().description("newDescr") - .build(), TableOption.fields(TableField.DESCRIPTION)); - assertTrue(updatedTableInfo.definition() instanceof StandardTableDefinition); - assertEquals(DATASET, updatedTableInfo.tableId().dataset()); - assertEquals(tableName, updatedTableInfo.tableId().table()); - assertEquals("newDescr", updatedTableInfo.description()); - assertNull(updatedTableInfo.definition().schema()); - assertNull(updatedTableInfo.lastModifiedTime()); - assertNull(updatedTableInfo.definition().numBytes()); - assertNull(updatedTableInfo.definition().numRows()); - assertTrue(bigquery.delete(DATASET, tableName)); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + Table updatedTable = bigquery.update(tableInfo.toBuilder().description("newDescr").build(), + TableOption.fields(TableField.DESCRIPTION)); + assertTrue(updatedTable.definition() instanceof StandardTableDefinition); + assertEquals(DATASET, updatedTable.tableId().dataset()); + assertEquals(tableName, updatedTable.tableId().table()); + assertEquals("newDescr", updatedTable.description()); + assertNull(updatedTable.definition().schema()); + assertNull(updatedTable.lastModifiedTime()); + assertNull(updatedTable.definition().numBytes()); + assertNull(updatedTable.definition().numRows()); + assertTrue(createdTable.delete()); } @Test @@ -544,14 +573,14 @@ public void testInsertAllWithSuffix() throws InterruptedException { assertFalse(response.hasErrors()); assertEquals(0, response.insertErrors().size()); String newTableName = tableName + "_suffix"; - TableInfo suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); + Table suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); // wait until the new table is created. If the table is never created the test will time-out while (suffixTable == null) { Thread.sleep(1000L); suffixTable = bigquery.getTable(DATASET, newTableName, TableOption.fields()); } assertTrue(bigquery.delete(TableId.of(DATASET, tableName))); - assertTrue(bigquery.delete(TableId.of(DATASET, newTableName))); + assertTrue(suffixTable.delete()); } @Test @@ -655,15 +684,15 @@ public void testQuery() throws InterruptedException { rowCount++; } assertEquals(2, rowCount); - JobInfo queryJob = bigquery.getJob(response.jobId()); + Job queryJob = bigquery.getJob(response.jobId()); JobStatistics.QueryStatistics statistics = queryJob.statistics(); assertNotNull(statistics.queryPlan()); } @Test public void testListJobs() { - Page jobs = bigquery.listJobs(); - for (JobInfo job : jobs.values()) { + Page jobs = bigquery.listJobs(); + for (Job job : jobs.values()) { assertNotNull(job.jobId()); assertNotNull(job.statistics()); assertNotNull(job.status()); @@ -674,8 +703,8 @@ public void testListJobs() { @Test public void testListJobsWithSelectedFields() { - Page jobs = bigquery.listJobs(JobListOption.fields(JobField.USER_EMAIL)); - for (JobInfo job : jobs.values()) { + Page jobs = bigquery.listJobs(JobListOption.fields(JobField.USER_EMAIL)); + for (Job job : jobs.values()) { assertNotNull(job.jobId()); assertNotNull(job.status()); assertNotNull(job.userEmail()); @@ -685,22 +714,21 @@ public void testListJobsWithSelectedFields() { } @Test - public void testCreateAndGetJob() throws InterruptedException { + public void testCreateAndGetJob() { String sourceTableName = "test_create_and_get_job_source_table"; String destinationTableName = "test_create_and_get_job_destination_table"; TableId sourceTable = TableId.of(DATASET, sourceTableName); StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); CopyJobConfiguration copyJobConfiguration = CopyJobConfiguration.of(destinationTable, sourceTable); - JobInfo job = JobInfo.of(copyJobConfiguration); - JobInfo createdJob = bigquery.create(job); - JobInfo remoteJob = bigquery.getJob(createdJob.jobId()); + Job createdJob = bigquery.create(JobInfo.of(copyJobConfiguration)); + Job remoteJob = bigquery.getJob(createdJob.jobId()); assertEquals(createdJob.jobId(), remoteJob.jobId()); CopyJobConfiguration createdConfiguration = createdJob.configuration(); CopyJobConfiguration remoteConfiguration = remoteJob.configuration(); @@ -713,25 +741,24 @@ public void testCreateAndGetJob() throws InterruptedException { assertNotNull(remoteJob.status()); assertEquals(createdJob.selfLink(), remoteJob.selfLink()); assertEquals(createdJob.userEmail(), remoteJob.userEmail()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); + assertTrue(createdTable.delete()); assertTrue(bigquery.delete(DATASET, destinationTableName)); } @Test - public void testCreateAndGetJobWithSelectedFields() throws InterruptedException { + public void testCreateAndGetJobWithSelectedFields() { String sourceTableName = "test_create_and_get_job_with_selected_fields_source_table"; String destinationTableName = "test_create_and_get_job_with_selected_fields_destination_table"; TableId sourceTable = TableId.of(DATASET, sourceTableName); StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, sourceTable); - JobInfo createdJob = - bigquery.create(JobInfo.of(configuration), JobOption.fields(JobField.ETAG)); + Job createdJob = bigquery.create(JobInfo.of(configuration), JobOption.fields(JobField.ETAG)); CopyJobConfiguration createdConfiguration = createdJob.configuration(); assertNotNull(createdJob.jobId()); assertNotNull(createdConfiguration.sourceTables()); @@ -741,7 +768,7 @@ public void testCreateAndGetJobWithSelectedFields() throws InterruptedException assertNull(createdJob.status()); assertNull(createdJob.selfLink()); assertNull(createdJob.userEmail()); - JobInfo remoteJob = bigquery.getJob(createdJob.jobId(), JobOption.fields(JobField.ETAG)); + Job remoteJob = bigquery.getJob(createdJob.jobId(), JobOption.fields(JobField.ETAG)); CopyJobConfiguration remoteConfiguration = remoteJob.configuration(); assertEquals(createdJob.jobId(), remoteJob.jobId()); assertEquals(createdConfiguration.sourceTables(), remoteConfiguration.sourceTables()); @@ -753,7 +780,7 @@ public void testCreateAndGetJobWithSelectedFields() throws InterruptedException assertNull(remoteJob.status()); assertNull(remoteJob.selfLink()); assertNull(remoteJob.userEmail()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); + assertTrue(createdTable.delete()); assertTrue(bigquery.delete(DATASET, destinationTableName)); } @@ -764,25 +791,24 @@ public void testCopyJob() throws InterruptedException { TableId sourceTable = TableId.of(DATASET, sourceTableName); StandardTableDefinition tableDefinition = StandardTableDefinition.of(TABLE_SCHEMA); TableInfo tableInfo = TableInfo.of(sourceTable, tableDefinition); - TableInfo createdTableInfo = bigquery.create(tableInfo); - assertNotNull(createdTableInfo); - assertEquals(DATASET, createdTableInfo.tableId().dataset()); - assertEquals(sourceTableName, createdTableInfo.tableId().table()); + Table createdTable = bigquery.create(tableInfo); + assertNotNull(createdTable); + assertEquals(DATASET, createdTable.tableId().dataset()); + assertEquals(sourceTableName, createdTable.tableId().table()); TableId destinationTable = TableId.of(DATASET, destinationTableName); CopyJobConfiguration configuration = CopyJobConfiguration.of(destinationTable, sourceTable); - JobInfo remoteJob = bigquery.create(JobInfo.of(configuration)); - while (remoteJob.status().state() != JobStatus.State.DONE) { + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); - TableInfo remoteTableInfo = bigquery.getTable(DATASET, destinationTableName); - assertNotNull(remoteTableInfo); - assertEquals(destinationTable.dataset(), remoteTableInfo.tableId().dataset()); - assertEquals(destinationTableName, remoteTableInfo.tableId().table()); - assertEquals(TABLE_SCHEMA, remoteTableInfo.definition().schema()); - assertTrue(bigquery.delete(DATASET, sourceTableName)); - assertTrue(bigquery.delete(DATASET, destinationTableName)); + Table remoteTable = bigquery.getTable(DATASET, destinationTableName); + assertNotNull(remoteTable); + assertEquals(destinationTable.dataset(), remoteTable.tableId().dataset()); + assertEquals(destinationTableName, remoteTable.tableId().table()); + assertEquals(TABLE_SCHEMA, remoteTable.definition().schema()); + assertTrue(createdTable.delete()); + assertTrue(remoteTable.delete()); } @Test @@ -797,10 +823,9 @@ public void testQueryJob() throws InterruptedException { .defaultDataset(DatasetId.of(DATASET)) .destinationTable(destinationTable) .build(); - JobInfo remoteJob = bigquery.create(JobInfo.of(configuration)); - while (remoteJob.status().state() != JobStatus.State.DONE) { + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); @@ -826,7 +851,7 @@ public void testQueryJob() throws InterruptedException { } assertEquals(2, rowCount); assertTrue(bigquery.delete(DATASET, tableName)); - JobInfo queryJob = bigquery.getJob(remoteJob.jobId()); + Job queryJob = bigquery.getJob(remoteJob.jobId()); JobStatistics.QueryStatistics statistics = queryJob.statistics(); assertNotNull(statistics.queryPlan()); } @@ -839,11 +864,9 @@ public void testExtractJob() throws InterruptedException { LoadJobConfiguration.builder(destinationTable, "gs://" + BUCKET + "/" + LOAD_FILE) .schema(SIMPLE_SCHEMA) .build(); - JobInfo remoteLoadJob = - bigquery.create(JobInfo.of(configuration)); - while (remoteLoadJob.status().state() != JobStatus.State.DONE) { + Job remoteLoadJob = bigquery.create(JobInfo.of(configuration)); + while (!remoteLoadJob.isDone()) { Thread.sleep(1000); - remoteLoadJob = bigquery.getJob(remoteLoadJob.jobId()); } assertNull(remoteLoadJob.status().error()); @@ -851,11 +874,9 @@ public void testExtractJob() throws InterruptedException { ExtractJobConfiguration.builder(destinationTable, "gs://" + BUCKET + "/" + EXTRACT_FILE) .printHeader(false) .build(); - JobInfo extractJob = JobInfo.of(extractConfiguration); - JobInfo remoteExtractJob = bigquery.create(extractJob); - while (remoteExtractJob.status().state() != JobStatus.State.DONE) { + Job remoteExtractJob = bigquery.create(JobInfo.of(extractConfiguration)); + while (!remoteExtractJob.isDone()) { Thread.sleep(1000); - remoteExtractJob = bigquery.getJob(remoteExtractJob.jobId()); } assertNull(remoteExtractJob.status().error()); assertEquals(CSV_CONTENT, @@ -872,22 +893,21 @@ public void testCancelJob() throws InterruptedException { .defaultDataset(DatasetId.of(DATASET)) .destinationTable(destinationTable) .build(); - JobInfo remoteJob = bigquery.create(JobInfo.of(configuration)); - assertTrue(bigquery.cancel(remoteJob.jobId())); - while (remoteJob.status().state() != JobStatus.State.DONE) { + Job remoteJob = bigquery.create(JobInfo.of(configuration)); + assertTrue(remoteJob.cancel()); + while (!remoteJob.isDone()) { Thread.sleep(1000); - remoteJob = bigquery.getJob(remoteJob.jobId()); } assertNull(remoteJob.status().error()); } @Test - public void testCancelNonExistingJob() throws InterruptedException { + public void testCancelNonExistingJob() { assertFalse(bigquery.cancel("test_cancel_non_existing_job")); } @Test - public void testInsertFromFile() throws InterruptedException, FileNotFoundException { + public void testInsertFromFile() throws InterruptedException { String destinationTableName = "test_insert_from_file_table"; TableId tableId = TableId.of(DATASET, destinationTableName); WriteChannelConfiguration configuration = WriteChannelConfiguration.builder(tableId) @@ -895,7 +915,7 @@ public void testInsertFromFile() throws InterruptedException, FileNotFoundExcept .createDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED) .schema(TABLE_SCHEMA) .build(); - try (TableDataWriteChannel channel = bigquery.writer(configuration)) { + try (WriteChannel channel = bigquery.writer(configuration)) { channel.write(ByteBuffer.wrap(JSON_CONTENT.getBytes(StandardCharsets.UTF_8))); } catch (IOException e) { fail("IOException was not expected"); diff --git a/gcloud-java-contrib/README.md b/gcloud-java-contrib/README.md index 23713f7450a3..b8bef6eb977e 100644 --- a/gcloud-java-contrib/README.md +++ b/gcloud-java-contrib/README.md @@ -3,38 +3,10 @@ Google Cloud Java Contributions Packages that provide higher-level abstraction/functionality for common gcloud-java use cases. -Quickstart ----------- -If you are using Maven, add this to your pom.xml file -```xml - - com.google.gcloud - gcloud-java-contrib - 0.1.3 - -``` -If you are using Gradle, add this to your dependencies -```Groovy -compile 'com.google.gcloud:gcloud-java-contrib:0.1.3' -``` -If you are using SBT, add this to your dependencies -```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-contrib" % "0.1.3" -``` - -Java Versions -------------- - -Java 7 or above is required for using this client. - -Versioning ----------- - -This library follows [Semantic Versioning] (http://semver.org/). - -It is currently in major version zero (``0.y.z``), which means that anything -may change at any time and the public API should not be considered -stable. +Contents +-------- + + * [gcloud-java-nio](./gcloud-java-nio/): NIO Filesystem Provider for Google Cloud Storage. Contributing ------------ diff --git a/gcloud-java-contrib/gcloud-java-nio/pom.xml b/gcloud-java-contrib/gcloud-java-nio/pom.xml new file mode 100644 index 000000000000..3f06179f30b7 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/pom.xml @@ -0,0 +1,117 @@ + + + 4.0.0 + com.google.gcloud + gcloud-java-nio + jar + GCloud Java NIO + + FileSystemProvider for Java NIO to access GCS transparently. + + + com.google.gcloud + gcloud-java-contrib + 0.1.4-SNAPSHOT + + + nio + + + + ${project.groupId} + gcloud-java + ${project.version} + + + com.google.guava + guava + 19.0 + + + com.google.code.findbugs + jsr305 + 2.0.1 + + + javax.inject + javax.inject + 1 + + + com.google.auto.service + auto-service + 1.0-rc2 + provided + + + com.google.auto.value + auto-value + 1.1 + provided + + + junit + junit + 4.12 + test + + + com.google.guava + guava-testlib + 19.0 + test + + + com.google.truth + truth + 0.27 + test + + + org.mockito + mockito-core + 1.9.5 + + + + + + org.codehaus.mojo + exec-maven-plugin + + false + + + + maven-compiler-plugin + 3.1 + + 1.7 + 1.7 + UTF-8 + -Xlint:unchecked + + + + maven-jar-plugin + 2.6 + + + true + true + + true + true + + + ${project.artifactId} + ${project.groupId} + ${project.version} + ${buildNumber} + + + + + + + diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfiguration.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfiguration.java new file mode 100644 index 000000000000..0371965e20ec --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfiguration.java @@ -0,0 +1,162 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; + +import com.google.auto.value.AutoValue; + +import java.util.Map; + +/** + * Configuration for {@link CloudStorageFileSystem} instances. + */ +@AutoValue +public abstract class CloudStorageConfiguration { + + /** + * Returns path of current working directory. This defaults to the root directory. + */ + public abstract String workingDirectory(); + + /** + * Returns {@code true} if we shouldn't throw an exception when encountering object names + * containing superfluous slashes, e.g. {@code a//b}. + */ + public abstract boolean permitEmptyPathComponents(); + + /** + * Returns {@code true} if '/' prefix on absolute object names should be removed before I/O. + * + *

If you disable this feature, please take into consideration that all paths created from a + * URI will have the leading slash. + */ + public abstract boolean stripPrefixSlash(); + + /** + * Returns {@code true} if paths with a trailing slash should be treated as fake directories. + */ + public abstract boolean usePseudoDirectories(); + + /** + * Returns block size (in bytes) used when talking to the GCS HTTP server. + */ + public abstract int blockSize(); + + /** + * Creates a new builder, initialized with the following settings: + * + *

    + *
  • Performing I/O on paths with extra slashes, e.g. {@code a//b} will throw an error. + *
  • The prefix slash on absolute paths will be removed when converting to an object name. + *
  • Pseudo-directories are enabled, so any path with a trailing slash is a fake directory. + *
+ */ + public static Builder builder() { + return new Builder(); + } + + /** + * Builder for {@link CloudStorageConfiguration}. + */ + public static final class Builder { + + private String workingDirectory = UnixPath.ROOT; + private boolean permitEmptyPathComponents = false; + private boolean stripPrefixSlash = true; + private boolean usePseudoDirectories = true; + private int blockSize = CloudStorageFileSystem.BLOCK_SIZE_DEFAULT; + + /** + * Changes current working directory for new filesystem. This cannot be changed once it's + * been set. You'll need to create another {@link CloudStorageFileSystem} object. + * + * @throws IllegalArgumentException if {@code path} is not absolute. + */ + public Builder workingDirectory(String path) { + checkArgument(UnixPath.getPath(false, path).isAbsolute(), "not absolute: %s", path); + workingDirectory = path; + return this; + } + + /** + * Configures whether or not we should throw an exception when encountering object names + * containing superfluous slashes, e.g. {@code a//b} + */ + public Builder permitEmptyPathComponents(boolean value) { + permitEmptyPathComponents = value; + return this; + } + + /** + * Configures if the '/' prefix on absolute object names should be removed before I/O. + * + *

If you disable this feature, please take into consideration that all paths created from a + * URI will have the leading slash. + */ + public Builder stripPrefixSlash(boolean value) { + stripPrefixSlash = value; + return this; + } + + /** + * Configures if paths with a trailing slash should be treated as fake directories. + */ + public Builder usePseudoDirectories(boolean value) { + usePseudoDirectories = value; + return this; + } + + /** + * Sets the block size in bytes that should be used for each HTTP request to the API. + * + *

The default is {@value CloudStorageFileSystem#BLOCK_SIZE_DEFAULT}. + */ + public Builder blockSize(int value) { + blockSize = value; + return this; + } + + /** + * Creates new instance without destroying builder. + */ + public CloudStorageConfiguration build() { + return new AutoValue_CloudStorageConfiguration( + workingDirectory, + permitEmptyPathComponents, + stripPrefixSlash, + usePseudoDirectories, + blockSize); + } + + Builder() {} + } + + static final CloudStorageConfiguration DEFAULT = builder().build(); + + static CloudStorageConfiguration fromMap(Map env) { + Builder builder = builder(); + for (Map.Entry entry : env.entrySet()) { + switch (entry.getKey()) { + case "workingDirectory": + builder.workingDirectory((String) entry.getValue()); + break; + case "permitEmptyPathComponents": + builder.permitEmptyPathComponents((Boolean) entry.getValue()); + break; + case "stripPrefixSlash": + builder.stripPrefixSlash((Boolean) entry.getValue()); + break; + case "usePseudoDirectories": + builder.usePseudoDirectories((Boolean) entry.getValue()); + break; + case "blockSize": + builder.blockSize((Integer) entry.getValue()); + break; + default: + throw new IllegalArgumentException(entry.getKey()); + } + } + return builder.build(); + } + + CloudStorageConfiguration() {} +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeView.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeView.java new file mode 100644 index 000000000000..49e7079d84ab --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeView.java @@ -0,0 +1,77 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.base.MoreObjects; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Storage; + +import java.io.IOException; +import java.nio.file.NoSuchFileException; +import java.nio.file.attribute.BasicFileAttributeView; +import java.nio.file.attribute.FileTime; +import java.util.Objects; + +import javax.annotation.concurrent.Immutable; + +/** + * Metadata view for a Google Cloud Storage object. + */ +@Immutable +public final class CloudStorageFileAttributeView implements BasicFileAttributeView { + + private final Storage storage; + private final CloudStoragePath path; + + CloudStorageFileAttributeView(Storage storage, CloudStoragePath path) { + this.storage = checkNotNull(storage); + this.path = checkNotNull(path); + } + + /** + * Returns {@value CloudStorageFileSystem#GCS_VIEW}. + */ + @Override + public String name() { + return CloudStorageFileSystem.GCS_VIEW; + } + + @Override + public CloudStorageFileAttributes readAttributes() throws IOException { + if (path.seemsLikeADirectory() && path.getFileSystem().config().usePseudoDirectories()) { + return new CloudStoragePseudoDirectoryAttributes(path); + } + BlobInfo blobInfo = storage.get(path.getBlobId()); + if (blobInfo == null) { + throw new NoSuchFileException(path.toUri().toString()); + } + + return new CloudStorageObjectAttributes(blobInfo); + } + + /** + * This feature is not supported, since Cloud Storage objects are immutable. + */ + @Override + public void setTimes(FileTime lastModifiedTime, FileTime lastAccessTime, FileTime createTime) { + throw new CloudStorageObjectImmutableException(); + } + + @Override + public boolean equals(Object other) { + return this == other + || other instanceof CloudStorageFileAttributeView + && Objects.equals(storage, ((CloudStorageFileAttributeView) other).storage) + && Objects.equals(path, ((CloudStorageFileAttributeView) other).path); + } + + @Override + public int hashCode() { + return Objects.hash(storage, path); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this).add("storage", storage).add("path", path).toString(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributes.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributes.java new file mode 100644 index 000000000000..61b7716455d6 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributes.java @@ -0,0 +1,63 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.common.base.Optional; +import com.google.common.collect.ImmutableMap; +import com.google.gcloud.storage.Acl; + +import java.nio.file.attribute.BasicFileAttributes; +import java.util.List; + +/** + * Interface for attributes on a Cloud Storage file or pseudo-directory. + */ +public interface CloudStorageFileAttributes extends BasicFileAttributes { + + /** + * Returns HTTP etag hash of object contents. + * + * @see "https://developers.google.com/storage/docs/hashes-etags" + */ + Optional etag(); + + /** + * Returns mime type (e.g. text/plain), if set. + * + * @see "http://en.wikipedia.org/wiki/Internet_media_type#List_of_common_media_types" + */ + Optional mimeType(); + + /** + * Returns access control list. + * + * @see "https://developers.google.com/storage/docs/reference-headers#acl" + */ + Optional> acl(); + + /** + * Returns {@code Cache-Control} HTTP header value, if set. + * + * @see "https://developers.google.com/storage/docs/reference-headers#cachecontrol" + */ + Optional cacheControl(); + + /** + * Returns {@code Content-Encoding} HTTP header value, if set. + * + * @see "https://developers.google.com/storage/docs/reference-headers#contentencoding" + */ + Optional contentEncoding(); + + /** + * Returns {@code Content-Disposition} HTTP header value, if set. + * + * @see "https://developers.google.com/storage/docs/reference-headers#contentdisposition" + */ + Optional contentDisposition(); + + /** + * Returns user-specified metadata. + * + * @see "https://developers.google.com/storage/docs/reference-headers#contentdisposition" + */ + ImmutableMap userMetadata(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystem.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystem.java new file mode 100644 index 000000000000..88f66150e123 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystem.java @@ -0,0 +1,208 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.collect.ImmutableSet; + +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.file.FileStore; +import java.nio.file.FileSystem; +import java.nio.file.Path; +import java.nio.file.PathMatcher; +import java.nio.file.WatchService; +import java.nio.file.attribute.FileTime; +import java.nio.file.attribute.UserPrincipalLookupService; +import java.util.Objects; +import java.util.Set; + +import javax.annotation.concurrent.Immutable; + +/** + * Google Cloud Storage {@link FileSystem} implementation. + * + * @see + * Concepts and Terminology + * @see + * Bucket and Object Naming Guidelines + */ +@Immutable +public final class CloudStorageFileSystem extends FileSystem { + + /** + * Returns Google Cloud Storage {@link FileSystem} object for {@code bucket}. + * + *

NOTE: You may prefer to use Java's standard API instead:

   {@code
+   *
+   *   FileSystem fs = FileSystems.getFileSystem(URI.create("gs://bucket"));}
+ * + *

However some systems and build environments might be flaky when it comes to Java SPI. This + * is because services are generally runtime dependencies and depend on a META-INF file being + * present in your jar (generated by Google Auto at compile-time). In such cases, this method + * provides a simpler alternative. + * + * @see #forBucket(String, CloudStorageConfiguration) + * @see java.nio.file.FileSystems#getFileSystem(java.net.URI) + */ + public static CloudStorageFileSystem forBucket(String bucket) { + return forBucket(bucket, CloudStorageConfiguration.DEFAULT); + } + + /** + * Creates new file system instance for {@code bucket}, with customizable settings. + * + * @see #forBucket(String) + */ + public static CloudStorageFileSystem forBucket(String bucket, CloudStorageConfiguration config) { + checkArgument( + !bucket.startsWith(URI_SCHEME + ":"), "Bucket name must not have schema: %s", bucket); + return new CloudStorageFileSystem( + new CloudStorageFileSystemProvider(), bucket, checkNotNull(config)); + } + + public static final String URI_SCHEME = "gs"; + public static final String GCS_VIEW = "gcs"; + public static final String BASIC_VIEW = "basic"; + public static final int BLOCK_SIZE_DEFAULT = 2 * 1024 * 1024; + public static final FileTime FILE_TIME_UNKNOWN = FileTime.fromMillis(0); + public static final ImmutableSet SUPPORTED_VIEWS = ImmutableSet.of(BASIC_VIEW, GCS_VIEW); + + private final CloudStorageFileSystemProvider provider; + private final String bucket; + private final CloudStorageConfiguration config; + + CloudStorageFileSystem( + CloudStorageFileSystemProvider provider, String bucket, CloudStorageConfiguration config) { + checkArgument(!bucket.isEmpty(), "bucket"); + this.provider = provider; + this.bucket = bucket; + this.config = config; + } + + @Override + public CloudStorageFileSystemProvider provider() { + return provider; + } + + /** + * Returns Cloud Storage bucket name being served by this file system. + */ + public String bucket() { + return bucket; + } + + /** + * Returns configuration object for this file system instance. + */ + public CloudStorageConfiguration config() { + return config; + } + + /** + * Converts Cloud Storage object name to a {@link Path} object. + */ + @Override + public CloudStoragePath getPath(String first, String... more) { + checkArgument( + !first.startsWith(URI_SCHEME + ":"), + "GCS FileSystem.getPath() must not have schema and bucket name: %s", + first); + return CloudStoragePath.getPath(this, first, more); + } + + /** + * Does nothing. + */ + @Override + public void close() {} + + /** + * Returns {@code true}. + */ + @Override + public boolean isOpen() { + return true; + } + + /** + * Returns {@code false}. + */ + @Override + public boolean isReadOnly() { + return false; + } + + /** + * Returns {@value UnixPath#SEPARATOR}. + */ + @Override + public String getSeparator() { + return "" + UnixPath.SEPARATOR; + } + + @Override + public Iterable getRootDirectories() { + return ImmutableSet.of(CloudStoragePath.getPath(this, UnixPath.ROOT)); + } + + @Override + public Iterable getFileStores() { + return ImmutableSet.of(); + } + + @Override + public Set supportedFileAttributeViews() { + return SUPPORTED_VIEWS; + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public PathMatcher getPathMatcher(String syntaxAndPattern) { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public UserPrincipalLookupService getUserPrincipalLookupService() { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public WatchService newWatchService() throws IOException { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + @Override + public boolean equals(Object other) { + return this == other + || other instanceof CloudStorageFileSystem + && Objects.equals(config, ((CloudStorageFileSystem) other).config) + && Objects.equals(bucket, ((CloudStorageFileSystem) other).bucket); + } + + @Override + public int hashCode() { + return Objects.hash(bucket); + } + + @Override + public String toString() { + try { + return new URI(URI_SCHEME, bucket, null, null).toString(); + } catch (URISyntaxException e) { + throw new AssertionError(e); + } + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProvider.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProvider.java new file mode 100644 index 000000000000..4c7acba98a96 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProvider.java @@ -0,0 +1,576 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; +import static com.google.common.base.Strings.isNullOrEmpty; +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.URI_SCHEME; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.checkBucket; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.checkNotNullArray; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.checkPath; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.stripPathFromUri; + +import com.google.auto.service.AutoService; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.MoreObjects; +import com.google.common.base.Throwables; +import com.google.common.primitives.Ints; +import com.google.gcloud.storage.Acl; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.CopyWriter; +import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.StorageException; +import com.google.gcloud.storage.StorageOptions; + +import java.io.BufferedInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.nio.channels.SeekableByteChannel; +import java.nio.file.AccessMode; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.CopyOption; +import java.nio.file.DirectoryStream; +import java.nio.file.DirectoryStream.Filter; +import java.nio.file.FileAlreadyExistsException; +import java.nio.file.FileStore; +import java.nio.file.LinkOption; +import java.nio.file.NoSuchFileException; +import java.nio.file.OpenOption; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.BasicFileAttributeView; +import java.nio.file.attribute.BasicFileAttributes; +import java.nio.file.attribute.FileAttribute; +import java.nio.file.attribute.FileAttributeView; +import java.nio.file.spi.FileSystemProvider; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +import javax.annotation.Nullable; +import javax.annotation.concurrent.ThreadSafe; + +/** + * Google Cloud Storage {@link FileSystemProvider} implementation. + */ +@ThreadSafe +@AutoService(FileSystemProvider.class) +public final class CloudStorageFileSystemProvider extends FileSystemProvider { + + private final Storage storage; + + // used only when we create a new instance of CloudStorageFileSystemProvider. + private static StorageOptions storageOptions; + + /** + * Sets options that are only used by the constructor. + */ + @VisibleForTesting + public static void setGCloudOptions(StorageOptions newStorageOptions) { + storageOptions = newStorageOptions; + } + + /** + * Default constructor which should only be called by Java SPI. + * + * @see java.nio.file.FileSystems#getFileSystem(URI) + * @see CloudStorageFileSystem#forBucket(String) + */ + public CloudStorageFileSystemProvider() { + this(storageOptions); + } + + private CloudStorageFileSystemProvider(@Nullable StorageOptions gcsStorageOptions) { + if (gcsStorageOptions == null) { + this.storage = StorageOptions.defaultInstance().service(); + } else { + this.storage = gcsStorageOptions.service(); + } + } + + @Override + public String getScheme() { + return URI_SCHEME; + } + + /** + * Returns Cloud Storage file system, provided a URI with no path, e.g. {@code gs://bucket}. + */ + @Override + public CloudStorageFileSystem getFileSystem(URI uri) { + return newFileSystem(uri, Collections.emptyMap()); + } + + /** + * Returns Cloud Storage file system, provided a URI with no path, e.g. {@code gs://bucket}. + */ + @Override + public CloudStorageFileSystem newFileSystem(URI uri, Map env) { + checkArgument( + uri.getScheme().equalsIgnoreCase(URI_SCHEME), + "Cloud Storage URIs must have '%s' scheme: %s", + URI_SCHEME, + uri); + checkArgument( + !isNullOrEmpty(uri.getHost()), "%s:// URIs must have a host: %s", URI_SCHEME, uri); + checkArgument( + uri.getPort() == -1 + && isNullOrEmpty(uri.getPath()) + && isNullOrEmpty(uri.getQuery()) + && isNullOrEmpty(uri.getFragment()) + && isNullOrEmpty(uri.getUserInfo()), + "GCS FileSystem URIs mustn't have: port, userinfo, path, query, or fragment: %s", + uri); + checkBucket(uri.getHost()); + return new CloudStorageFileSystem(this, uri.getHost(), CloudStorageConfiguration.fromMap(env)); + } + + @Override + public CloudStoragePath getPath(URI uri) { + return CloudStoragePath.getPath(getFileSystem(stripPathFromUri(uri)), uri.getPath()); + } + + @Override + public SeekableByteChannel newByteChannel( + Path path, Set options, FileAttribute... attrs) throws IOException { + checkNotNull(path); + checkNotNullArray(attrs); + if (options.contains(StandardOpenOption.WRITE)) { + // TODO: Make our OpenOptions implement FileAttribute. Also remove buffer option. + return newWriteChannel(path, options); + } else { + return newReadChannel(path, options); + } + } + + private SeekableByteChannel newReadChannel(Path path, Set options) + throws IOException { + for (OpenOption option : options) { + if (option instanceof StandardOpenOption) { + switch ((StandardOpenOption) option) { + case READ: + // Default behavior. + break; + case SPARSE: + case TRUNCATE_EXISTING: + // Ignored by specification. + break; + case WRITE: + throw new IllegalArgumentException("READ+WRITE not supported yet"); + case APPEND: + case CREATE: + case CREATE_NEW: + case DELETE_ON_CLOSE: + case DSYNC: + case SYNC: + default: + throw new UnsupportedOperationException(option.toString()); + } + } else { + throw new UnsupportedOperationException(option.toString()); + } + } + CloudStoragePath cloudPath = checkPath(path); + if (cloudPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + throw new CloudStoragePseudoDirectoryException(cloudPath); + } + return CloudStorageReadChannel.create(storage, cloudPath.getBlobId(), 0); + } + + private SeekableByteChannel newWriteChannel(Path path, Set options) + throws IOException { + + CloudStoragePath cloudPath = checkPath(path); + if (cloudPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + throw new CloudStoragePseudoDirectoryException(cloudPath); + } + BlobId file = cloudPath.getBlobId(); + BlobInfo.Builder infoBuilder = BlobInfo.builder(file); + List writeOptions = new ArrayList<>(); + List acls = new ArrayList<>(); + + HashMap metas = new HashMap<>(); + for (OpenOption option : options) { + if (option instanceof OptionMimeType) { + infoBuilder.contentType(((OptionMimeType) option).mimeType()); + } else if (option instanceof OptionCacheControl) { + infoBuilder.cacheControl(((OptionCacheControl) option).cacheControl()); + } else if (option instanceof OptionContentDisposition) { + infoBuilder.contentDisposition(((OptionContentDisposition) option).contentDisposition()); + } else if (option instanceof OptionContentEncoding) { + infoBuilder.contentEncoding(((OptionContentEncoding) option).contentEncoding()); + } else if (option instanceof OptionUserMetadata) { + OptionUserMetadata opMeta = (OptionUserMetadata) option; + metas.put(opMeta.key(), opMeta.value()); + } else if (option instanceof OptionAcl) { + acls.add(((OptionAcl) option).acl()); + } else if (option instanceof OptionBlockSize) { + // TODO: figure out how to plumb in block size. + } else if (option instanceof StandardOpenOption) { + switch ((StandardOpenOption) option) { + case CREATE: + case TRUNCATE_EXISTING: + case WRITE: + // Default behavior. + break; + case SPARSE: + // Ignored by specification. + break; + case CREATE_NEW: + writeOptions.add(Storage.BlobWriteOption.doesNotExist()); + break; + case READ: + throw new IllegalArgumentException("READ+WRITE not supported yet"); + case APPEND: + case DELETE_ON_CLOSE: + case DSYNC: + case SYNC: + default: + throw new UnsupportedOperationException(option.toString()); + } + } else if (option instanceof CloudStorageOption) { + // XXX: We need to interpret these later + } else { + throw new UnsupportedOperationException(option.toString()); + } + } + + if (!metas.isEmpty()) { + infoBuilder.metadata(metas); + } + if (!acls.isEmpty()) { + infoBuilder.acl(acls); + } + + try { + return new CloudStorageWriteChannel( + storage.writer( + infoBuilder.build(), writeOptions.toArray(new Storage.BlobWriteOption[0]))); + } catch (StorageException oops) { + throw asIOException(oops); + } + } + + @Override + public InputStream newInputStream(Path path, OpenOption... options) throws IOException { + InputStream result = super.newInputStream(path, options); + CloudStoragePath cloudPath = checkPath(path); + int blockSize = cloudPath.getFileSystem().config().blockSize(); + for (OpenOption option : options) { + if (option instanceof OptionBlockSize) { + blockSize = ((OptionBlockSize) option).size(); + } + } + return new BufferedInputStream(result, blockSize); + } + + @Override + public boolean deleteIfExists(Path path) throws IOException { + CloudStoragePath cloudPath = checkPath(path); + if (cloudPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + throw new CloudStoragePseudoDirectoryException(cloudPath); + } + return storage.delete(cloudPath.getBlobId()); + } + + @Override + public void delete(Path path) throws IOException { + CloudStoragePath cloudPath = checkPath(path); + if (!deleteIfExists(cloudPath)) { + throw new NoSuchFileException(cloudPath.toString()); + } + } + + @Override + public void move(Path source, Path target, CopyOption... options) throws IOException { + for (CopyOption option : options) { + if (option == StandardCopyOption.ATOMIC_MOVE) { + throw new AtomicMoveNotSupportedException( + source.toString(), + target.toString(), + "Google Cloud Storage does not support atomic move operations."); + } + } + copy(source, target, options); + delete(source); + } + + @Override + public void copy(Path source, Path target, CopyOption... options) throws IOException { + boolean wantCopyAttributes = false; + boolean wantReplaceExisting = false; + boolean setContentType = false; + boolean setCacheControl = false; + boolean setContentEncoding = false; + boolean setContentDisposition = false; + + CloudStoragePath toPath = checkPath(target); + BlobInfo.Builder tgtInfoBuilder = BlobInfo.builder(toPath.getBlobId()).contentType(""); + + int blockSize = -1; + for (CopyOption option : options) { + if (option instanceof StandardCopyOption) { + switch ((StandardCopyOption) option) { + case COPY_ATTRIBUTES: + wantCopyAttributes = true; + break; + case REPLACE_EXISTING: + wantReplaceExisting = true; + break; + case ATOMIC_MOVE: + default: + throw new UnsupportedOperationException(option.toString()); + } + } else if (option instanceof CloudStorageOption) { + if (option instanceof OptionBlockSize) { + blockSize = ((OptionBlockSize) option).size(); + } else if (option instanceof OptionMimeType) { + tgtInfoBuilder.contentType(((OptionMimeType) option).mimeType()); + setContentType = true; + } else if (option instanceof OptionCacheControl) { + tgtInfoBuilder.cacheControl(((OptionCacheControl) option).cacheControl()); + setCacheControl = true; + } else if (option instanceof OptionContentEncoding) { + tgtInfoBuilder.contentEncoding(((OptionContentEncoding) option).contentEncoding()); + setContentEncoding = true; + } else if (option instanceof OptionContentDisposition) { + tgtInfoBuilder.contentDisposition( + ((OptionContentDisposition) option).contentDisposition()); + setContentDisposition = true; + } else { + throw new UnsupportedOperationException(option.toString()); + } + } else { + throw new UnsupportedOperationException(option.toString()); + } + } + + CloudStoragePath fromPath = checkPath(source); + + blockSize = + blockSize != -1 + ? blockSize + : Ints.max( + fromPath.getFileSystem().config().blockSize(), + toPath.getFileSystem().config().blockSize()); + // TODO: actually use blockSize + + if (fromPath.seemsLikeADirectory() && toPath.seemsLikeADirectory()) { + if (fromPath.getFileSystem().config().usePseudoDirectories() + && toPath.getFileSystem().config().usePseudoDirectories()) { + // NOOP: This would normally create an empty directory. + return; + } else { + checkArgument( + !fromPath.getFileSystem().config().usePseudoDirectories() + && !toPath.getFileSystem().config().usePseudoDirectories(), + "File systems associated with paths don't agree on pseudo-directories."); + } + } + if (fromPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + throw new CloudStoragePseudoDirectoryException(fromPath); + } + if (toPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + throw new CloudStoragePseudoDirectoryException(toPath); + } + + try { + if (wantCopyAttributes) { + BlobInfo blobInfo = storage.get(fromPath.getBlobId()); + if (null == blobInfo) { + throw new NoSuchFileException(fromPath.toString()); + } + if (!setCacheControl) { + tgtInfoBuilder.cacheControl(blobInfo.cacheControl()); + } + if (!setContentType) { + tgtInfoBuilder.contentType(blobInfo.contentType()); + } + if (!setContentEncoding) { + tgtInfoBuilder.contentEncoding(blobInfo.contentEncoding()); + } + if (!setContentDisposition) { + tgtInfoBuilder.contentDisposition(blobInfo.contentDisposition()); + } + tgtInfoBuilder.acl(blobInfo.acl()); + tgtInfoBuilder.metadata(blobInfo.metadata()); + } + + BlobInfo tgtInfo = tgtInfoBuilder.build(); + Storage.CopyRequest.Builder copyReqBuilder = + Storage.CopyRequest.builder().source(fromPath.getBlobId()); + if (wantReplaceExisting) { + copyReqBuilder = copyReqBuilder.target(tgtInfo); + } else { + copyReqBuilder = copyReqBuilder.target(tgtInfo, Storage.BlobTargetOption.doesNotExist()); + } + CopyWriter copyWriter = storage.copy(copyReqBuilder.build()); + copyWriter.result(); + } catch (StorageException oops) { + throw asIOException(oops); + } + } + + @Override + public boolean isSameFile(Path path, Path path2) { + return checkPath(path).equals(checkPath(path2)); + } + + /** + * Always returns {@code false}, because GCS doesn't support hidden files. + */ + @Override + public boolean isHidden(Path path) { + checkPath(path); + return false; + } + + @Override + public void checkAccess(Path path, AccessMode... modes) throws IOException { + for (AccessMode mode : modes) { + switch (mode) { + case READ: + case WRITE: + break; + case EXECUTE: + default: + throw new UnsupportedOperationException(mode.toString()); + } + } + CloudStoragePath cloudPath = checkPath(path); + if (cloudPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + return; + } + if (storage.get(cloudPath.getBlobId(), Storage.BlobGetOption.fields(Storage.BlobField.ID)) + == null) { + throw new NoSuchFileException(path.toString()); + } + } + + @Override + public A readAttributes( + Path path, Class type, LinkOption... options) throws IOException { + checkNotNull(type); + checkNotNullArray(options); + if (type != CloudStorageFileAttributes.class && type != BasicFileAttributes.class) { + throw new UnsupportedOperationException(type.getSimpleName()); + } + CloudStoragePath cloudPath = checkPath(path); + if (cloudPath.seemsLikeADirectoryAndUsePseudoDirectories()) { + @SuppressWarnings("unchecked") + A result = (A) new CloudStoragePseudoDirectoryAttributes(cloudPath); + return result; + } + BlobInfo blobInfo = storage.get(cloudPath.getBlobId()); + // null size indicate a file that we haven't closed yet, so GCS treats it as not there yet. + if (null == blobInfo || blobInfo.size() == null) { + throw new NoSuchFileException( + cloudPath.getBlobId().bucket() + "/" + cloudPath.getBlobId().name()); + } + CloudStorageObjectAttributes ret; + ret = new CloudStorageObjectAttributes(blobInfo); + @SuppressWarnings("unchecked") + A result = (A) ret; + return result; + } + + @Override + public Map readAttributes(Path path, String attributes, LinkOption... options) { + // Java 7 NIO defines at least eleven string attributes we'd want to support + // (eg. BasicFileAttributeView and PosixFileAttributeView), so rather than a partial + // implementation we rely on the other overload for now. + throw new UnsupportedOperationException(); + } + + @Override + public V getFileAttributeView( + Path path, Class type, LinkOption... options) { + checkNotNull(type); + checkNotNullArray(options); + if (type != CloudStorageFileAttributeView.class && type != BasicFileAttributeView.class) { + throw new UnsupportedOperationException(type.getSimpleName()); + } + CloudStoragePath cloudPath = checkPath(path); + @SuppressWarnings("unchecked") + V result = (V) new CloudStorageFileAttributeView(storage, cloudPath); + return result; + } + + /** + * Does nothing since GCS uses fake directories. + */ + @Override + public void createDirectory(Path dir, FileAttribute... attrs) { + checkPath(dir); + checkNotNullArray(attrs); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public DirectoryStream newDirectoryStream(Path dir, Filter filter) { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + /** + * Throws {@link UnsupportedOperationException} because Cloud Storage objects are immutable. + */ + @Override + public void setAttribute(Path path, String attribute, Object value, LinkOption... options) { + throw new CloudStorageObjectImmutableException(); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public FileStore getFileStore(Path path) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean equals(Object other) { + return this == other + || other instanceof CloudStorageFileSystemProvider + && Objects.equals(storage, ((CloudStorageFileSystemProvider) other).storage); + } + + @Override + public int hashCode() { + return Objects.hash(storage); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this).add("storage", storage).toString(); + } + + private IOException asIOException(StorageException oops) { + if (oops.code() == 404) { + return new NoSuchFileException(oops.reason()); + } + // TODO: research if other codes should be translated to IOException. + + // RPC API can only throw StorageException, but CloudStorageFileSystemProvider + // can only throw IOException. Square peg, round hole. + Throwable cause = oops.getCause(); + try { + if (cause instanceof FileAlreadyExistsException) { + throw new FileAlreadyExistsException(((FileAlreadyExistsException) cause).getReason()); + } + // fallback + Throwables.propagateIfInstanceOf(oops.getCause(), IOException.class); + } catch (IOException okEx) { + return okEx; + } + return new IOException(oops.getMessage(), oops); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectAttributes.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectAttributes.java new file mode 100644 index 000000000000..405b507bdad1 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectAttributes.java @@ -0,0 +1,133 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkNotNull; +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.FILE_TIME_UNKNOWN; + +import com.google.common.base.MoreObjects; +import com.google.common.base.Optional; +import com.google.common.collect.ImmutableMap; +import com.google.gcloud.storage.Acl; +import com.google.gcloud.storage.BlobInfo; + +import java.nio.file.attribute.FileTime; +import java.util.List; +import java.util.Objects; + +import javax.annotation.Nonnull; +import javax.annotation.concurrent.Immutable; + +/** + * Metadata for a Google Cloud Storage file. + */ +@Immutable +final class CloudStorageObjectAttributes implements CloudStorageFileAttributes { + + @Nonnull private final BlobInfo info; + + CloudStorageObjectAttributes(BlobInfo info) { + this.info = checkNotNull(info); + } + + @Override + public long size() { + return info.size(); + } + + @Override + public FileTime creationTime() { + if (info.updateTime() == null) { + return FILE_TIME_UNKNOWN; + } + return FileTime.fromMillis(info.updateTime()); + } + + @Override + public FileTime lastModifiedTime() { + return creationTime(); + } + + @Override + public Optional etag() { + return Optional.fromNullable(info.etag()); + } + + @Override + public Optional mimeType() { + return Optional.fromNullable(info.contentType()); + } + + @Override + public Optional> acl() { + return Optional.fromNullable(info.acl()); + } + + @Override + public Optional cacheControl() { + return Optional.fromNullable(info.cacheControl()); + } + + @Override + public Optional contentEncoding() { + return Optional.fromNullable(info.contentEncoding()); + } + + @Override + public Optional contentDisposition() { + return Optional.fromNullable(info.contentDisposition()); + } + + @Override + public ImmutableMap userMetadata() { + if (null == info.metadata()) { + return ImmutableMap.of(); + } + return ImmutableMap.copyOf(info.metadata()); + } + + @Override + public boolean isDirectory() { + return false; + } + + @Override + public boolean isOther() { + return false; + } + + @Override + public boolean isRegularFile() { + return true; + } + + @Override + public boolean isSymbolicLink() { + return false; + } + + @Override + public FileTime lastAccessTime() { + return FILE_TIME_UNKNOWN; + } + + @Override + public Object fileKey() { + return info.id(); + } + + @Override + public boolean equals(Object other) { + return this == other + || other instanceof CloudStorageObjectAttributes + && Objects.equals(info, ((CloudStorageObjectAttributes) other).info); + } + + @Override + public int hashCode() { + return info.hashCode(); + } + + @Override + public String toString() { + return MoreObjects.toStringHelper(this).add("info", info).toString(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectImmutableException.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectImmutableException.java new file mode 100644 index 000000000000..a97eec9c72ee --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageObjectImmutableException.java @@ -0,0 +1,11 @@ +package com.google.gcloud.storage.contrib.nio; + +/** + * Exception reminding user that Cloud Storage objects can't be mutated. + */ +public final class CloudStorageObjectImmutableException extends UnsupportedOperationException { + + CloudStorageObjectImmutableException() { + super("Cloud Storage objects are immutable."); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOption.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOption.java new file mode 100644 index 000000000000..f662561f0276 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOption.java @@ -0,0 +1,25 @@ +package com.google.gcloud.storage.contrib.nio; + +import java.nio.file.CopyOption; +import java.nio.file.OpenOption; + +/** + * Master interface for file operation option classes related to Google Cloud Storage. + */ +public interface CloudStorageOption { + + /** + * Interface for GCS options that can be specified when opening files. + */ + interface Open extends CloudStorageOption, OpenOption {} + + /** + * Interface for GCS options that can be specified when copying files. + */ + interface Copy extends CloudStorageOption, CopyOption {} + + /** + * Interface for GCS options that can be specified when opening or copying files. + */ + interface OpenCopy extends Open, Copy {} +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptions.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptions.java new file mode 100644 index 000000000000..7417dcb1467c --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptions.java @@ -0,0 +1,79 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.gcloud.storage.Acl; + +/** + * Helper class for specifying options when opening and copying Cloud Storage files. + */ +public final class CloudStorageOptions { + + /** + * Sets the mime type header on an object, e.g. {@code "text/plain"}. + */ + public static CloudStorageOption.OpenCopy withMimeType(String mimeType) { + return OptionMimeType.create(mimeType); + } + + /** + * Disables caching on an object. Same as: {@code withCacheControl("no-cache")}. + */ + public static CloudStorageOption.OpenCopy withoutCaching() { + return withCacheControl("no-cache"); + } + + /** + * Sets the {@code Cache-Control} HTTP header on an object. + * + * @see "https://developers.google.com/storage/docs/reference-headers#cachecontrol" + */ + public static CloudStorageOption.OpenCopy withCacheControl(String cacheControl) { + return OptionCacheControl.create(cacheControl); + } + + /** + * Sets the {@code Content-Disposition} HTTP header on an object. + * + * @see "https://developers.google.com/storage/docs/reference-headers#contentdisposition" + */ + public static CloudStorageOption.OpenCopy withContentDisposition(String contentDisposition) { + return OptionContentDisposition.create(contentDisposition); + } + + /** + * Sets the {@code Content-Encoding} HTTP header on an object. + * + * @see "https://developers.google.com/storage/docs/reference-headers#contentencoding" + */ + public static CloudStorageOption.OpenCopy withContentEncoding(String contentEncoding) { + return OptionContentEncoding.create(contentEncoding); + } + + /** + * Sets the ACL value on a Cloud Storage object. + * + * @see "https://developers.google.com/storage/docs/reference-headers#acl" + */ + public static CloudStorageOption.OpenCopy withAcl(Acl acl) { + return OptionAcl.create(acl); + } + + /** + * Sets an unmodifiable piece of user metadata on a Cloud Storage object. + * + * @see "https://developers.google.com/storage/docs/reference-headers#xgoogmeta" + */ + public static CloudStorageOption.OpenCopy withUserMetadata(String key, String value) { + return OptionUserMetadata.create(key, value); + } + + /** + * Sets the block size (in bytes) when talking to the GCS server. + * + *

The default is {@value CloudStorageFileSystem#BLOCK_SIZE_DEFAULT}. + */ + public static CloudStorageOption.OpenCopy withBlockSize(int size) { + return OptionBlockSize.create(size); + } + + private CloudStorageOptions() {} +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePath.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePath.java new file mode 100644 index 000000000000..5fff71a4d7ee --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePath.java @@ -0,0 +1,339 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.URI_SCHEME; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.checkNotNullArray; +import static com.google.gcloud.storage.contrib.nio.CloudStorageUtil.checkPath; + +import com.google.common.collect.UnmodifiableIterator; +import com.google.gcloud.storage.BlobId; + +import java.io.File; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.file.LinkOption; +import java.nio.file.Path; +import java.nio.file.WatchEvent.Kind; +import java.nio.file.WatchEvent.Modifier; +import java.nio.file.WatchKey; +import java.nio.file.WatchService; +import java.util.Collections; +import java.util.Iterator; +import java.util.Objects; +import java.util.regex.Pattern; + +import javax.annotation.Nullable; +import javax.annotation.concurrent.Immutable; + +/** + * Google Cloud Storage {@link Path}. + * + * @see UnixPath + */ +@Immutable +public final class CloudStoragePath implements Path { + + private static final Pattern EXTRA_SLASHES_OR_DOT_DIRS_PATTERN = + Pattern.compile("^\\.\\.?/|//|/\\.\\.?/|/\\.\\.?$"); + + private final CloudStorageFileSystem fileSystem; + private final UnixPath path; + + private CloudStoragePath(CloudStorageFileSystem fileSystem, UnixPath path) { + this.fileSystem = fileSystem; + this.path = path; + } + + static CloudStoragePath getPath(CloudStorageFileSystem fileSystem, String path, String... more) { + return new CloudStoragePath( + fileSystem, UnixPath.getPath(fileSystem.config().permitEmptyPathComponents(), path, more)); + } + + /** + * Returns the Cloud Storage bucket name being served by this file system. + */ + public String bucket() { + return fileSystem.bucket(); + } + + /** + * Returns path converted to a {@link BlobId} so I/O can be performed. + */ + BlobId getBlobId() { + return BlobId.of(bucket(), toRealPath().path.toString()); + } + + boolean seemsLikeADirectory() { + return path.seemsLikeADirectory(); + } + + boolean seemsLikeADirectoryAndUsePseudoDirectories() { + return path.seemsLikeADirectory() && fileSystem.config().usePseudoDirectories(); + } + + @Override + public CloudStorageFileSystem getFileSystem() { + return fileSystem; + } + + @Nullable + @Override + public CloudStoragePath getRoot() { + return newPath(path.getRoot()); + } + + @Override + public boolean isAbsolute() { + return path.isAbsolute(); + } + + /** + * Changes relative path to be absolute, using + * {@link CloudStorageConfiguration#workingDirectory() workingDirectory} as current dir. + */ + @Override + public CloudStoragePath toAbsolutePath() { + return newPath(path.toAbsolutePath(getWorkingDirectory())); + } + + /** + * Returns this path rewritten to the Cloud Storage object name that'd be used to perform i/o. + * + *

This method makes path {@link #toAbsolutePath() absolute} and removes the prefix slash from + * the absolute path when {@link CloudStorageConfiguration#stripPrefixSlash() stripPrefixSlash} + * is {@code true}. + * + * @throws IllegalArgumentException if path contains extra slashes or dot-dirs when + * {@link CloudStorageConfiguration#permitEmptyPathComponents() permitEmptyPathComponents} + * is {@code false}, or if the resulting path is empty. + */ + @Override + public CloudStoragePath toRealPath(LinkOption... options) { + checkNotNullArray(options); + return newPath(toRealPathInternal(true)); + } + + private UnixPath toRealPathInternal(boolean errorCheck) { + UnixPath objectName = path.toAbsolutePath(getWorkingDirectory()); + if (errorCheck && !fileSystem.config().permitEmptyPathComponents()) { + checkArgument( + !EXTRA_SLASHES_OR_DOT_DIRS_PATTERN.matcher(objectName).find(), + "I/O not allowed on dot-dirs or extra slashes when !permitEmptyPathComponents: %s", + objectName); + } + if (fileSystem.config().stripPrefixSlash()) { + objectName = objectName.removeBeginningSeparator(); + } + checkArgument( + !errorCheck || !objectName.isEmpty(), "I/O not allowed on empty GCS object names."); + return objectName; + } + + /** + * Returns path without extra slashes or {@code .} and {@code ..} and preserves trailing slash. + */ + @Override + public CloudStoragePath normalize() { + return newPath(path.normalize()); + } + + @Override + public CloudStoragePath resolve(Path object) { + return newPath(path.resolve(checkPath(object).path)); + } + + @Override + public CloudStoragePath resolve(String other) { + return newPath(path.resolve(getUnixPath(other))); + } + + @Override + public CloudStoragePath resolveSibling(Path other) { + return newPath(path.resolveSibling(checkPath(other).path)); + } + + @Override + public CloudStoragePath resolveSibling(String other) { + return newPath(path.resolveSibling(getUnixPath(other))); + } + + @Override + public CloudStoragePath relativize(Path object) { + return newPath(path.relativize(checkPath(object).path)); + } + + @Nullable + @Override + public CloudStoragePath getParent() { + return newPath(path.getParent()); + } + + @Nullable + @Override + public CloudStoragePath getFileName() { + return newPath(path.getFileName()); + } + + @Override + public CloudStoragePath subpath(int beginIndex, int endIndex) { + return newPath(path.subpath(beginIndex, endIndex)); + } + + @Override + public int getNameCount() { + return path.getNameCount(); + } + + @Override + public CloudStoragePath getName(int index) { + return newPath(path.getName(index)); + } + + @Override + public boolean startsWith(Path other) { + if (!(checkNotNull(other) instanceof CloudStoragePath)) { + return false; + } + CloudStoragePath that = (CloudStoragePath) other; + if (!bucket().equals(that.bucket())) { + return false; + } + return path.startsWith(that.path); + } + + @Override + public boolean startsWith(String other) { + return path.startsWith(getUnixPath(other)); + } + + @Override + public boolean endsWith(Path other) { + if (!(checkNotNull(other) instanceof CloudStoragePath)) { + return false; + } + CloudStoragePath that = (CloudStoragePath) other; + if (!bucket().equals(that.bucket())) { + return false; + } + return path.endsWith(that.path); + } + + @Override + public boolean endsWith(String other) { + return path.endsWith(getUnixPath(other)); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public WatchKey register(WatchService watcher, Kind[] events, Modifier... modifiers) { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + /** + * Throws {@link UnsupportedOperationException} because this feature hasn't been implemented yet. + */ + @Override + public WatchKey register(WatchService watcher, Kind... events) { + // TODO: Implement me. + throw new UnsupportedOperationException(); + } + + /** + * Throws {@link UnsupportedOperationException} because GCS files are not backed by the local file + * system. + */ + @Override + public File toFile() { + throw new UnsupportedOperationException("GCS objects aren't available locally"); + } + + @Override + public Iterator iterator() { + if (path.isEmpty()) { + return Collections.singleton(this).iterator(); + } else if (path.isRoot()) { + return Collections.emptyIterator(); + } else { + return new PathIterator(); + } + } + + @Override + public int compareTo(Path other) { + // Documented to throw CCE if other is associated with a different FileSystemProvider. + CloudStoragePath that = (CloudStoragePath) other; + int res = bucket().compareTo(that.bucket()); + if (res != 0) { + return res; + } + return toRealPathInternal(false).compareTo(that.toRealPathInternal(false)); + } + + @Override + public boolean equals(Object other) { + return this == other + || other instanceof CloudStoragePath + && Objects.equals(bucket(), ((CloudStoragePath) other).bucket()) + && Objects.equals( + toRealPathInternal(false), ((CloudStoragePath) other).toRealPathInternal(false)); + } + + @Override + public int hashCode() { + return Objects.hash(bucket(), toRealPathInternal(false)); + } + + @Override + public String toString() { + return path.toString(); + } + + @Override + public URI toUri() { + try { + return new URI(URI_SCHEME, bucket(), path.toAbsolutePath().toString(), null); + } catch (URISyntaxException e) { + throw new AssertionError(e); + } + } + + @Nullable + private CloudStoragePath newPath(@Nullable UnixPath newPath) { + if (newPath == path) { // Nonuse of equals is intentional. + return this; + } else if (newPath != null) { + return new CloudStoragePath(fileSystem, newPath); + } else { + return null; + } + } + + private UnixPath getUnixPath(String newPath) { + return UnixPath.getPath(fileSystem.config().permitEmptyPathComponents(), newPath); + } + + private UnixPath getWorkingDirectory() { + return getUnixPath(fileSystem.config().workingDirectory()); + } + + /** + * Transform iterator providing a slight performance boost over {@code FluentIterable}. + */ + private final class PathIterator extends UnmodifiableIterator { + private final Iterator delegate = path.split(); + + @Override + public Path next() { + return newPath(getUnixPath(delegate.next())); + } + + @Override + public boolean hasNext() { + return delegate.hasNext(); + } + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryAttributes.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryAttributes.java new file mode 100644 index 000000000000..cdd3191563d2 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryAttributes.java @@ -0,0 +1,102 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.FILE_TIME_UNKNOWN; + +import com.google.common.base.Optional; +import com.google.common.collect.ImmutableMap; +import com.google.gcloud.storage.Acl; + +import java.nio.file.attribute.FileTime; +import java.util.List; + +/** + * Metadata for a Cloud Storage pseudo-directory. + */ +final class CloudStoragePseudoDirectoryAttributes implements CloudStorageFileAttributes { + + private final String id; + + CloudStoragePseudoDirectoryAttributes(CloudStoragePath path) { + this.id = path.toUri().toString(); + } + + @Override + public boolean isDirectory() { + return true; + } + + @Override + public boolean isOther() { + return false; + } + + @Override + public boolean isRegularFile() { + return false; + } + + @Override + public boolean isSymbolicLink() { + return false; + } + + @Override + public Object fileKey() { + return id; + } + + @Override + public long size() { + return 1; // Allow I/O to happen before we fail. + } + + @Override + public FileTime lastModifiedTime() { + return FILE_TIME_UNKNOWN; + } + + @Override + public FileTime creationTime() { + return FILE_TIME_UNKNOWN; + } + + @Override + public FileTime lastAccessTime() { + return FILE_TIME_UNKNOWN; + } + + @Override + public Optional etag() { + return Optional.absent(); + } + + @Override + public Optional mimeType() { + return Optional.absent(); + } + + @Override + public Optional> acl() { + return Optional.absent(); + } + + @Override + public Optional cacheControl() { + return Optional.absent(); + } + + @Override + public Optional contentEncoding() { + return Optional.absent(); + } + + @Override + public Optional contentDisposition() { + return Optional.absent(); + } + + @Override + public ImmutableMap userMetadata() { + return ImmutableMap.of(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryException.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryException.java new file mode 100644 index 000000000000..281b7257a1a6 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStoragePseudoDirectoryException.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import java.nio.file.InvalidPathException; + +/** + * Exception thrown when erroneously trying to operate on a path with a trailing slash. + */ +public final class CloudStoragePseudoDirectoryException extends InvalidPathException { + + CloudStoragePseudoDirectoryException(CloudStoragePath path) { + super(path.toString(), "Can't perform I/O on pseudo-directories (trailing slash)"); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannel.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannel.java new file mode 100644 index 000000000000..544f7c89d7f2 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannel.java @@ -0,0 +1,134 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; + +import com.google.gcloud.ReadChannel; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Storage; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonWritableChannelException; +import java.nio.channels.SeekableByteChannel; +import java.nio.file.NoSuchFileException; + +import javax.annotation.CheckReturnValue; +import javax.annotation.concurrent.ThreadSafe; + +/** + * Cloud Storage read channel. + * + * @see CloudStorageWriteChannel + */ +@ThreadSafe +final class CloudStorageReadChannel implements SeekableByteChannel { + + @CheckReturnValue + @SuppressWarnings("resource") + static CloudStorageReadChannel create(Storage gcsStorage, BlobId file, long position) + throws IOException { + // XXX: Reading size and opening file should be atomic. + long size = fetchSize(gcsStorage, file); + ReadChannel channel = gcsStorage.reader(file); + if (position > 0) { + channel.seek((int) position); + } + return new CloudStorageReadChannel(position, size, channel); + } + + private final ReadChannel channel; + private long position; + private long size; + + private CloudStorageReadChannel(long position, long size, ReadChannel channel) { + this.position = position; + this.size = size; + this.channel = channel; + } + + @Override + public boolean isOpen() { + synchronized (this) { + return channel.isOpen(); + } + } + + @Override + public void close() throws IOException { + synchronized (this) { + channel.close(); + } + } + + @Override + public int read(ByteBuffer dst) throws IOException { + synchronized (this) { + checkOpen(); + int amt = channel.read(dst); + if (amt > 0) { + position += amt; + // XXX: This would only ever happen if the fetchSize() race-condition occurred. + if (position > size) { + size = position; + } + } + return amt; + } + } + + @Override + public long size() throws IOException { + synchronized (this) { + checkOpen(); + return size; + } + } + + @Override + public long position() throws IOException { + synchronized (this) { + checkOpen(); + return position; + } + } + + @Override + public SeekableByteChannel position(long newPosition) throws IOException { + checkArgument(newPosition >= 0); + synchronized (this) { + checkOpen(); + if (newPosition == position) { + return this; + } + channel.seek((int) newPosition); + position = newPosition; + return this; + } + } + + @Override + public int write(ByteBuffer src) throws IOException { + throw new NonWritableChannelException(); + } + + @Override + public SeekableByteChannel truncate(long size) throws IOException { + throw new NonWritableChannelException(); + } + + private void checkOpen() throws ClosedChannelException { + if (!channel.isOpen()) { + throw new ClosedChannelException(); + } + } + + private static long fetchSize(Storage gcsStorage, BlobId file) throws IOException { + BlobInfo blobInfo = gcsStorage.get(file); + if (blobInfo == null) { + throw new NoSuchFileException(String.format("gs://%s/%s", file.bucket(), file.name())); + } + return blobInfo.size(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageUtil.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageUtil.java new file mode 100644 index 000000000000..d342c2a2c8b9 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageUtil.java @@ -0,0 +1,64 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.file.Path; +import java.nio.file.ProviderMismatchException; +import java.util.regex.Pattern; + +final class CloudStorageUtil { + + private static final Pattern BUCKET_PATTERN = Pattern.compile("[a-z0-9][-._a-z0-9]+[a-z0-9]"); + + static void checkBucket(String bucket) { + // TODO: The true check is actually more complicated. Consider implementing it. + checkArgument( + BUCKET_PATTERN.matcher(bucket).matches(), + "Invalid bucket name: '" + + bucket + + "'. " + + "GCS bucket names must contain only lowercase letters, numbers, dashes (-), " + + "underscores (_), and dots (.). Bucket names must start and end with a number or a " + + "letter. See the following page for more details: " + + "https://developers.google.com/storage/docs/bucketnaming"); + } + + static CloudStoragePath checkPath(Path path) { + if (!(checkNotNull(path) instanceof CloudStoragePath)) { + throw new ProviderMismatchException( + String.format( + "Not a Cloud Storage path: %s (%s)", path, path.getClass().getSimpleName())); + } + return (CloudStoragePath) path; + } + + static URI stripPathFromUri(URI uri) { + try { + return new URI( + uri.getScheme(), + uri.getUserInfo(), + uri.getHost(), + uri.getPort(), + null, + uri.getQuery(), + uri.getFragment()); + } catch (URISyntaxException e) { + throw new IllegalArgumentException(e.getMessage()); + } + } + + /** + * Makes {@code NullPointerTester} happy. + */ + @SafeVarargs + static void checkNotNullArray(T... values) { + for (T value : values) { + checkNotNull(value); + } + } + + private CloudStorageUtil() {} +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannel.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannel.java new file mode 100644 index 000000000000..92c95fb688d5 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannel.java @@ -0,0 +1,96 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.gcloud.WriteChannel; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonReadableChannelException; +import java.nio.channels.SeekableByteChannel; + +import javax.annotation.concurrent.ThreadSafe; + +/** + * Cloud Storage write channel. + * + *

This class does not support seeking, reading, or append. + * + * @see CloudStorageReadChannel + */ +@ThreadSafe +final class CloudStorageWriteChannel implements SeekableByteChannel { + + private final WriteChannel channel; + private long position; + private long size; + + CloudStorageWriteChannel(WriteChannel channel) { + this.channel = channel; + } + + @Override + public boolean isOpen() { + synchronized (this) { + return channel.isOpen(); + } + } + + @Override + public void close() throws IOException { + synchronized (this) { + channel.close(); + } + } + + @Override + public int read(ByteBuffer dst) throws IOException { + throw new NonReadableChannelException(); + } + + @Override + public int write(ByteBuffer src) throws IOException { + synchronized (this) { + checkOpen(); + int amt = channel.write(src); + if (amt > 0) { + position += amt; + size += amt; + } + return amt; + } + } + + @Override + public long position() throws IOException { + synchronized (this) { + checkOpen(); + return position; + } + } + + @Override + public SeekableByteChannel position(long newPosition) throws IOException { + throw new UnsupportedOperationException(); + } + + @Override + public long size() throws IOException { + synchronized (this) { + checkOpen(); + return size; + } + } + + @Override + public SeekableByteChannel truncate(long newSize) throws IOException { + // TODO: Emulate this functionality by closing and rewriting old file up to newSize. + // Or maybe just swap out GcsStorage for the API client. + throw new UnsupportedOperationException(); + } + + private void checkOpen() throws ClosedChannelException { + if (!channel.isOpen()) { + throw new ClosedChannelException(); + } + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionAcl.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionAcl.java new file mode 100644 index 000000000000..2224cdacebb7 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionAcl.java @@ -0,0 +1,14 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; +import com.google.gcloud.storage.Acl; + +@AutoValue +abstract class OptionAcl implements CloudStorageOption.OpenCopy { + + static OptionAcl create(Acl acl) { + return new AutoValue_OptionAcl(acl); + } + + abstract Acl acl(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionBlockSize.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionBlockSize.java new file mode 100644 index 000000000000..f33f601fad61 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionBlockSize.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionBlockSize implements CloudStorageOption.OpenCopy { + + static OptionBlockSize create(int size) { + return new AutoValue_OptionBlockSize(size); + } + + abstract int size(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionCacheControl.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionCacheControl.java new file mode 100644 index 000000000000..c33d4394fa24 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionCacheControl.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionCacheControl implements CloudStorageOption.OpenCopy { + + static OptionCacheControl create(String cacheControl) { + return new AutoValue_OptionCacheControl(cacheControl); + } + + abstract String cacheControl(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentDisposition.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentDisposition.java new file mode 100644 index 000000000000..7bebefcba19c --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentDisposition.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionContentDisposition implements CloudStorageOption.OpenCopy { + + static OptionContentDisposition create(String contentDisposition) { + return new AutoValue_OptionContentDisposition(contentDisposition); + } + + abstract String contentDisposition(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentEncoding.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentEncoding.java new file mode 100644 index 000000000000..879d05983f37 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionContentEncoding.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionContentEncoding implements CloudStorageOption.OpenCopy { + + static OptionContentEncoding create(String contentEncoding) { + return new AutoValue_OptionContentEncoding(contentEncoding); + } + + abstract String contentEncoding(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionMimeType.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionMimeType.java new file mode 100644 index 000000000000..feb1b0c6dd1b --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionMimeType.java @@ -0,0 +1,13 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionMimeType implements CloudStorageOption.OpenCopy { + + static OptionMimeType create(String mimeType) { + return new AutoValue_OptionMimeType(mimeType); + } + + abstract String mimeType(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionUserMetadata.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionUserMetadata.java new file mode 100644 index 000000000000..af8ea0306a93 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/OptionUserMetadata.java @@ -0,0 +1,15 @@ +package com.google.gcloud.storage.contrib.nio; + +import com.google.auto.value.AutoValue; + +@AutoValue +abstract class OptionUserMetadata implements CloudStorageOption.OpenCopy { + + static OptionUserMetadata create(String key, String value) { + return new AutoValue_OptionUserMetadata(key, value); + } + + abstract String key(); + + abstract String value(); +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/UnixPath.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/UnixPath.java new file mode 100644 index 000000000000..c603e4342b51 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/UnixPath.java @@ -0,0 +1,525 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.base.Joiner; +import com.google.common.base.Splitter; +import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; +import com.google.common.collect.Ordering; +import com.google.common.collect.PeekingIterator; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; + +import javax.annotation.Nullable; +import javax.annotation.concurrent.Immutable; + +/** + * Unix file system path. + * + *

This class is helpful for writing {@link java.nio.file.Path Path} implementations. + * + *

This implementation behaves almost identically to {@code sun.nio.fs.UnixPath}. The only + * difference is that some methods (like {@link #relativize(UnixPath)} go to greater lengths to + * preserve trailing backslashes, in order to ensure the path will continue to be recognized as a + * directory. + * + *

Note: This code might not play nice with + * Supplementary + * Characters as Surrogates. + */ +@Immutable +final class UnixPath implements CharSequence { + + public static final char DOT = '.'; + public static final char SEPARATOR = '/'; + public static final String ROOT = "" + SEPARATOR; + public static final String CURRENT_DIR = "" + DOT; + public static final String PARENT_DIR = "" + DOT + DOT; + public static final UnixPath EMPTY_PATH = new UnixPath(false, ""); + public static final UnixPath ROOT_PATH = new UnixPath(false, ROOT); + + private static final Splitter SPLITTER = Splitter.on(SEPARATOR).omitEmptyStrings(); + private static final Splitter SPLITTER_PERMIT_EMPTY_COMPONENTS = Splitter.on(SEPARATOR); + private static final Joiner JOINER = Joiner.on(SEPARATOR); + private static final Ordering> ORDERING = Ordering.natural().lexicographical(); + + private final String path; + private List lazyStringParts; + private final boolean permitEmptyComponents; + + private UnixPath(boolean permitEmptyComponents, String path) { + this.path = checkNotNull(path); + this.permitEmptyComponents = permitEmptyComponents; + } + + /** + * Returns new path of {@code first}. + */ + public static UnixPath getPath(boolean permitEmptyComponents, String path) { + if (path.isEmpty()) { + return EMPTY_PATH; + } else if (isRootInternal(path)) { + return ROOT_PATH; + } else { + return new UnixPath(permitEmptyComponents, path); + } + } + + /** + * Returns new path of {@code first} with {@code more} components resolved against it. + * + * @see #resolve(UnixPath) + * @see java.nio.file.FileSystem#getPath(String, String...) + */ + public static UnixPath getPath(boolean permitEmptyComponents, String first, String... more) { + if (more.length == 0) { + return getPath(permitEmptyComponents, first); + } + StringBuilder builder = new StringBuilder(first); + for (int i = 0; i < more.length; i++) { + String part = more[i]; + if (part.isEmpty()) { + continue; + } else if (isAbsoluteInternal(part)) { + if (i == more.length - 1) { + return new UnixPath(permitEmptyComponents, part); + } else { + builder.replace(0, builder.length(), part); + } + } else if (hasTrailingSeparatorInternal(builder)) { + builder.append(part); + } else { + builder.append(SEPARATOR); + builder.append(part); + } + } + return new UnixPath(permitEmptyComponents, builder.toString()); + } + + /** + * Returns {@code true} consists only of {@code separator}. + */ + public boolean isRoot() { + return isRootInternal(path); + } + + private static boolean isRootInternal(String path) { + return path.length() == 1 && path.charAt(0) == SEPARATOR; + } + + /** + * Returns {@code true} if path starts with {@code separator}. + */ + public boolean isAbsolute() { + return isAbsoluteInternal(path); + } + + private static boolean isAbsoluteInternal(String path) { + return !path.isEmpty() && path.charAt(0) == SEPARATOR; + } + + /** + * Returns {@code true} if path ends with {@code separator}. + */ + public boolean hasTrailingSeparator() { + return hasTrailingSeparatorInternal(path); + } + + private static boolean hasTrailingSeparatorInternal(CharSequence path) { + return path.length() != 0 && path.charAt(path.length() - 1) == SEPARATOR; + } + + /** + * Returns {@code true} if path ends with a trailing slash, or would after normalization. + */ + public boolean seemsLikeADirectory() { + int length = path.length(); + return path.isEmpty() + || path.charAt(length - 1) == SEPARATOR + || path.endsWith(".") && (length == 1 || path.charAt(length - 2) == SEPARATOR) + || path.endsWith("..") && (length == 2 || path.charAt(length - 3) == SEPARATOR); + } + + /** + * Returns last component in {@code path}. + * + * @see java.nio.file.Path#getFileName() + */ + @Nullable + public UnixPath getFileName() { + if (path.isEmpty()) { + return EMPTY_PATH; + } else if (isRoot()) { + return null; + } else { + List parts = getParts(); + String last = parts.get(parts.size() - 1); + return parts.size() == 1 && path.equals(last) + ? this + : new UnixPath(permitEmptyComponents, last); + } + } + + /** + * Returns parent directory (including trailing separator) or {@code null} if no parent remains. + * + * @see java.nio.file.Path#getParent() + */ + @Nullable + public UnixPath getParent() { + if (path.isEmpty() || isRoot()) { + return null; + } + int index = + hasTrailingSeparator() + ? path.lastIndexOf(SEPARATOR, path.length() - 2) + : path.lastIndexOf(SEPARATOR); + if (index == -1) { + return isAbsolute() ? ROOT_PATH : null; + } else { + return new UnixPath(permitEmptyComponents, path.substring(0, index + 1)); + } + } + + /** + * Returns root component if an absolute path, otherwise {@code null}. + * + * @see java.nio.file.Path#getRoot() + */ + @Nullable + public UnixPath getRoot() { + return isAbsolute() ? ROOT_PATH : null; + } + + /** + * Returns specified range of sub-components in path joined together. + * + * @see java.nio.file.Path#subpath(int, int) + */ + public UnixPath subpath(int beginIndex, int endIndex) { + if (path.isEmpty() && beginIndex == 0 && endIndex == 1) { + return this; + } + checkArgument(beginIndex >= 0 && endIndex > beginIndex); + List subList; + try { + subList = getParts().subList(beginIndex, endIndex); + } catch (IndexOutOfBoundsException e) { + throw new IllegalArgumentException(); + } + return new UnixPath(permitEmptyComponents, JOINER.join(subList)); + } + + /** + * Returns number of components in {@code path}. + * + * @see java.nio.file.Path#getNameCount() + */ + public int getNameCount() { + if (path.isEmpty()) { + return 1; + } else if (isRoot()) { + return 0; + } else { + return getParts().size(); + } + } + + /** + * Returns component in {@code path} at {@code index}. + * + * @see java.nio.file.Path#getName(int) + */ + public UnixPath getName(int index) { + if (path.isEmpty()) { + return this; + } + try { + return new UnixPath(permitEmptyComponents, getParts().get(index)); + } catch (IndexOutOfBoundsException e) { + throw new IllegalArgumentException(); + } + } + + /** + * Returns path without extra separators or {@code .} and {@code ..}, preserving trailing slash. + * + * @see java.nio.file.Path#normalize() + */ + public UnixPath normalize() { + List parts = new ArrayList<>(); + boolean mutated = false; + int resultLength = 0; + int mark = 0; + int index; + do { + index = path.indexOf(SEPARATOR, mark); + String part = path.substring(mark, index == -1 ? path.length() : index + 1); + switch (part) { + case CURRENT_DIR: + case CURRENT_DIR + SEPARATOR: + mutated = true; + break; + case PARENT_DIR: + case PARENT_DIR + SEPARATOR: + mutated = true; + if (!parts.isEmpty()) { + resultLength -= parts.remove(parts.size() - 1).length(); + } + break; + default: + if (index != mark || index == 0) { + parts.add(part); + resultLength = part.length(); + } else { + mutated = true; + } + } + mark = index + 1; + } while (index != -1); + if (!mutated) { + return this; + } + StringBuilder result = new StringBuilder(resultLength); + for (String part : parts) { + result.append(part); + } + return new UnixPath(permitEmptyComponents, result.toString()); + } + + /** + * Returns {@code other} appended to {@code path}. + * + * @see java.nio.file.Path#resolve(java.nio.file.Path) + */ + public UnixPath resolve(UnixPath other) { + if (other.path.isEmpty()) { + return this; + } else if (other.isAbsolute()) { + return other; + } else if (hasTrailingSeparator()) { + return new UnixPath(permitEmptyComponents, path + other.path); + } else { + return new UnixPath(permitEmptyComponents, path + SEPARATOR + other.path); + } + } + + /** + * Returns {@code other} resolved against parent of {@code path}. + * + * @see java.nio.file.Path#resolveSibling(java.nio.file.Path) + */ + public UnixPath resolveSibling(UnixPath other) { + checkNotNull(other); + UnixPath parent = getParent(); + return parent == null ? other : parent.resolve(other); + } + + /** + * Returns {@code other} made relative to {@code path}. + * + * @see java.nio.file.Path#relativize(java.nio.file.Path) + */ + public UnixPath relativize(UnixPath other) { + checkArgument(isAbsolute() == other.isAbsolute(), "'other' is different type of Path"); + if (path.isEmpty()) { + return other; + } + PeekingIterator left = Iterators.peekingIterator(split()); + PeekingIterator right = Iterators.peekingIterator(other.split()); + while (left.hasNext() && right.hasNext()) { + if (!left.peek().equals(right.peek())) { + break; + } + left.next(); + right.next(); + } + StringBuilder result = new StringBuilder(path.length() + other.path.length()); + while (left.hasNext()) { + result.append(PARENT_DIR); + result.append(SEPARATOR); + left.next(); + } + while (right.hasNext()) { + result.append(right.next()); + result.append(SEPARATOR); + } + if (result.length() > 0 && !other.hasTrailingSeparator()) { + result.deleteCharAt(result.length() - 1); + } + return new UnixPath(permitEmptyComponents, result.toString()); + } + + /** + * Returns {@code true} if {@code path} starts with {@code other}. + * + * @see java.nio.file.Path#startsWith(java.nio.file.Path) + */ + public boolean startsWith(UnixPath other) { + UnixPath me = removeTrailingSeparator(); + other = other.removeTrailingSeparator(); + if (other.path.length() > me.path.length()) { + return false; + } else if (me.isAbsolute() != other.isAbsolute()) { + return false; + } else if (!me.path.isEmpty() && other.path.isEmpty()) { + return false; + } + return startsWith(split(), other.split()); + } + + private static boolean startsWith(Iterator lefts, Iterator rights) { + while (rights.hasNext()) { + if (!lefts.hasNext() || !rights.next().equals(lefts.next())) { + return false; + } + } + return true; + } + + /** + * Returns {@code true} if {@code path} ends with {@code other}. + * + * @see java.nio.file.Path#endsWith(java.nio.file.Path) + */ + public boolean endsWith(UnixPath other) { + UnixPath me = removeTrailingSeparator(); + other = other.removeTrailingSeparator(); + if (other.path.length() > me.path.length()) { + return false; + } else if (!me.path.isEmpty() && other.path.isEmpty()) { + return false; + } else if (other.isAbsolute()) { + return me.isAbsolute() && me.path.equals(other.path); + } + return startsWith(me.splitReverse(), other.splitReverse()); + } + + /** + * Compares two paths lexicographically for ordering. + * + * @see java.nio.file.Path#compareTo(java.nio.file.Path) + */ + public int compareTo(UnixPath other) { + return ORDERING.compare(getParts(), other.getParts()); + } + + /** + * Converts relative path to an absolute path. + */ + public UnixPath toAbsolutePath(UnixPath currentWorkingDirectory) { + checkArgument(currentWorkingDirectory.isAbsolute()); + return isAbsolute() ? this : currentWorkingDirectory.resolve(this); + } + + /** + * Returns {@code toAbsolutePath(ROOT_PATH)}. + */ + public UnixPath toAbsolutePath() { + return toAbsolutePath(ROOT_PATH); + } + + /** + * Removes beginning separator from path, if an absolute path. + */ + public UnixPath removeBeginningSeparator() { + return isAbsolute() ? new UnixPath(permitEmptyComponents, path.substring(1)) : this; + } + + /** + * Adds trailing separator to path, if it isn't present. + */ + public UnixPath addTrailingSeparator() { + return hasTrailingSeparator() ? this : new UnixPath(permitEmptyComponents, path + SEPARATOR); + } + + /** + * Removes trailing separator from path, unless it's root. + */ + public UnixPath removeTrailingSeparator() { + if (!isRoot() && hasTrailingSeparator()) { + return new UnixPath(permitEmptyComponents, path.substring(0, path.length() - 1)); + } else { + return this; + } + } + + /** + * Splits path into components, excluding separators and empty strings. + */ + public Iterator split() { + return getParts().iterator(); + } + + /** + * Splits path into components in reverse, excluding separators and empty strings. + */ + public Iterator splitReverse() { + return Lists.reverse(getParts()).iterator(); + } + + @Override + public boolean equals(Object other) { + return this == other || other instanceof UnixPath && path.equals(((UnixPath) other).path); + } + + @Override + public int hashCode() { + return path.hashCode(); + } + + /** + * Returns path as a string. + */ + @Override + public String toString() { + return path; + } + + @Override + public int length() { + return path.length(); + } + + @Override + public char charAt(int index) { + return path.charAt(index); + } + + @Override + public CharSequence subSequence(int start, int end) { + return path.subSequence(start, end); + } + + /** + * Returns {@code true} if this path is an empty string. + */ + public boolean isEmpty() { + return path.isEmpty(); + } + + /** + * Returns list of path components, excluding slashes. + */ + private List getParts() { + List result = lazyStringParts; + return result != null + ? result + : (lazyStringParts = + path.isEmpty() || isRoot() ? Collections.emptyList() : createParts()); + } + + private List createParts() { + if (permitEmptyComponents) { + return SPLITTER_PERMIT_EMPTY_COMPONENTS.splitToList( + path.charAt(0) == SEPARATOR ? path.substring(1) : path); + } else { + return SPLITTER.splitToList(path); + } + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/package-info.java b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/package-info.java new file mode 100644 index 000000000000..78980d92dbe2 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/main/java/com/google/gcloud/storage/contrib/nio/package-info.java @@ -0,0 +1,82 @@ +/** + * Java 7 nio FileSystem client library for Google Cloud Storage. + * + *

This client library allows you to easily interact with Google Cloud Storage, using Java's + * standard file system API, introduced in Java 7. + * + *

How It Works

+ * + * The simplest way to get started is with {@code Paths} and {@code Files}:
   {@code
+ *
+ *   Path path = Paths.get(URI.create("gs://bucket/lolcat.csv"));
+ *   List lines = Files.readAllLines(path, StandardCharsets.UTF_8);}
+ * + *

If you want to configure the bucket per-environment, it might make more sense to use the + * {@code FileSystem} API: + *

+ *   class Foo {
+ *     static String bucket = System.getProperty(...);
+ *     static FileSystem fs = FileSystems.getFileSystem(URI.create("gs://" + bucket));
+ *     void bar() {
+ *       byte[] data = "hello world".getBytes(StandardCharsets.UTF_8);
+ *       Path path = fs.getPath("/object");
+ *       Files.write(path, data);
+ *       data = Files.readBytes(path);
+ *     }
+ *     void doodle() {
+ *       Path path = fs.getPath("/path/to/doodle.csv");
+ *       List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);
+ *     }
+ *   }
+ * + *

You can also use InputStream and OutputStream for streaming: + *

+ *   Path path = Paths.get(URI.create("gs://bucket/lolcat.csv"));
+ *   try (InputStream input = Files.openInputStream(path)) {
+ *     // ...
+ *   }
+ * + *

You can set various attributes using + * {@link com.google.gcloud.storage.contrib.nio.CloudStorageOptions CloudStorageOptions} static + * helpers: + *

+ *   Files.write(csvPath, csvLines, StandardCharsets.UTF_8,
+ *       withMimeType(MediaType.CSV_UTF8),
+ *       withoutCaching());
+ * + *

NOTE: Cloud Storage uses a flat namespace and therefore doesn't support real + * directories. So this library supports what's known as "pseudo-directories". Any path that + * includes a trailing slash, will be considered a directory. It will always be assumed to exist, + * without performing any I/O. This allows you to do path manipulation in the same manner as you + * would with the normal UNIX file system implementation. You can disable this feature with + * {@link com.google.gcloud.storage.contrib.nio.CloudStorageConfiguration#usePseudoDirectories()}. + * + *

Unit Testing

+ * + *

Here's a simple unit test:

+ *
+ *   class MyTest {
+ *     {@literal @}Rule
+ *     public final AppEngineRule appEngine = AppEngineRule.builder().build();
+ *
+ *     {@literal @}Test
+ *     public test_fileWrite() throws Exception {
+ *       Path path = Paths.get(URI.create("gs://bucket/traditional"));
+ *       Files.write(path, "eyebrow".getBytes(StandardCharsets.US_ASCII));
+ *       assertEquals("eyebrow", new String(Files.readBytes(path), StandardCharsets.US_ASCII));
+ *     }
+ *   }
+ * + *

Non-SPI Interface

+ * + *

If you don't want to rely on Java SPI, which requires a META-INF file in your jar generated by + * Google Auto, you can instantiate this file system directly as follows:

   {@code
+ *
+ *   CloudStorageFileSystem fs = CloudStorageFileSystemProvider.forBucket("bucket");
+ *   byte[] data = "hello world".getBytes(StandardCharsets.UTF_8);
+ *   Path path = fs.getPath("/object");
+ *   Files.write(path, data);
+ *   data = Files.readBytes(path);}
+ */ +@javax.annotation.ParametersAreNonnullByDefault +package com.google.gcloud.storage.contrib.nio; diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfigurationTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfigurationTest.java new file mode 100644 index 000000000000..66963b18ddb9 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageConfigurationTest.java @@ -0,0 +1,61 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; + +import com.google.common.collect.ImmutableMap; + +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +/** + * Unit tests for {@link CloudStorageConfiguration}. + */ +@RunWith(JUnit4.class) +public class CloudStorageConfigurationTest { + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + @Test + public void testBuilder() { + CloudStorageConfiguration config = + CloudStorageConfiguration.builder() + .workingDirectory("/omg") + .permitEmptyPathComponents(true) + .stripPrefixSlash(false) + .usePseudoDirectories(false) + .blockSize(666) + .build(); + assertThat(config.workingDirectory()).isEqualTo("/omg"); + assertThat(config.permitEmptyPathComponents()).isTrue(); + assertThat(config.stripPrefixSlash()).isFalse(); + assertThat(config.usePseudoDirectories()).isFalse(); + assertThat(config.blockSize()).isEqualTo(666); + } + + @Test + public void testFromMap() { + CloudStorageConfiguration config = + CloudStorageConfiguration.fromMap( + new ImmutableMap.Builder() + .put("workingDirectory", "/omg") + .put("permitEmptyPathComponents", true) + .put("stripPrefixSlash", false) + .put("usePseudoDirectories", false) + .put("blockSize", 666) + .build()); + assertThat(config.workingDirectory()).isEqualTo("/omg"); + assertThat(config.permitEmptyPathComponents()).isTrue(); + assertThat(config.stripPrefixSlash()).isFalse(); + assertThat(config.usePseudoDirectories()).isFalse(); + assertThat(config.blockSize()).isEqualTo(666); + } + + @Test + public void testFromMap_badKey_throwsIae() { + thrown.expect(IllegalArgumentException.class); + CloudStorageConfiguration.fromMap(ImmutableMap.of("lol", "/omg")); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeViewTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeViewTest.java new file mode 100644 index 000000000000..71e35ab1745f --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributeViewTest.java @@ -0,0 +1,99 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withCacheControl; +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.common.testing.EqualsTester; +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.attribute.FileTime; + +/** + * Unit tests for {@link CloudStorageFileAttributeView}. + */ +@RunWith(JUnit4.class) +public class CloudStorageFileAttributeViewTest { + + private static final byte[] HAPPY = "(✿◕ ‿◕ )ノ".getBytes(UTF_8); + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + private Path path; + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + path = Paths.get(URI.create("gs://red/water")); + } + + @Test + public void testReadAttributes() throws IOException { + Files.write(path, HAPPY, withCacheControl("potato")); + CloudStorageFileAttributeView lazyAttributes = + Files.getFileAttributeView(path, CloudStorageFileAttributeView.class); + assertThat(lazyAttributes.readAttributes().cacheControl().get()).isEqualTo("potato"); + } + + @Test + public void testReadAttributes_notFound_throwsNoSuchFileException() throws IOException { + CloudStorageFileAttributeView lazyAttributes = + Files.getFileAttributeView(path, CloudStorageFileAttributeView.class); + thrown.expect(NoSuchFileException.class); + lazyAttributes.readAttributes(); + } + + @Test + public void testReadAttributes_pseudoDirectory() throws IOException { + Path dir = Paths.get(URI.create("gs://red/rum/")); + CloudStorageFileAttributeView lazyAttributes = + Files.getFileAttributeView(dir, CloudStorageFileAttributeView.class); + assertThat(lazyAttributes.readAttributes()) + .isInstanceOf(CloudStoragePseudoDirectoryAttributes.class); + } + + @Test + public void testName() throws IOException { + Files.write(path, HAPPY, withCacheControl("potato")); + CloudStorageFileAttributeView lazyAttributes = + Files.getFileAttributeView(path, CloudStorageFileAttributeView.class); + assertThat(lazyAttributes.name()).isEqualTo("gcs"); + } + + @Test + public void testEquals_equalsTester() { + new EqualsTester() + .addEqualityGroup( + Files.getFileAttributeView( + Paths.get(URI.create("gs://red/rum")), CloudStorageFileAttributeView.class), + Files.getFileAttributeView( + Paths.get(URI.create("gs://red/rum")), CloudStorageFileAttributeView.class)) + .addEqualityGroup( + Files.getFileAttributeView( + Paths.get(URI.create("gs://red/lol/dog")), CloudStorageFileAttributeView.class)) + .testEquals(); + } + + @Test + public void testNullness() throws NoSuchMethodException, SecurityException { + new NullPointerTester() + .ignore(CloudStorageFileAttributeView.class.getMethod("equals", Object.class)) + .setDefault(FileTime.class, FileTime.fromMillis(0)) + .testAllPublicInstanceMethods( + Files.getFileAttributeView(path, CloudStorageFileAttributeView.class)); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributesTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributesTest.java new file mode 100644 index 000000000000..92f9acd1407b --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileAttributesTest.java @@ -0,0 +1,169 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withAcl; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withCacheControl; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentDisposition; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentEncoding; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withMimeType; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withUserMetadata; +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.common.testing.EqualsTester; +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.Acl; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** + * Unit tests for {@link CloudStorageFileAttributes}. + */ +@RunWith(JUnit4.class) +public class CloudStorageFileAttributesTest { + + private static final byte[] HAPPY = "(✿◕ ‿◕ )ノ".getBytes(UTF_8); + + private Path path; + private Path dir; + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + path = Paths.get(URI.create("gs://bucket/randompath")); + dir = Paths.get(URI.create("gs://bucket/randompath/")); + } + + @Test + public void testCacheControl() throws IOException { + Files.write(path, HAPPY, withCacheControl("potato")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).cacheControl().get()) + .isEqualTo("potato"); + } + + @Test + public void testMimeType() throws IOException { + Files.write(path, HAPPY, withMimeType("text/potato")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).mimeType().get()) + .isEqualTo("text/potato"); + } + + @Test + public void testAcl() throws IOException { + Acl acl = Acl.of(new Acl.User("serf@example.com"), Acl.Role.READER); + Files.write(path, HAPPY, withAcl(acl)); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).acl().get()) + .contains(acl); + } + + @Test + public void testContentDisposition() throws IOException { + Files.write(path, HAPPY, withContentDisposition("crash call")); + assertThat( + Files.readAttributes(path, CloudStorageFileAttributes.class).contentDisposition().get()) + .isEqualTo("crash call"); + } + + @Test + public void testContentEncoding() throws IOException { + Files.write(path, HAPPY, withContentEncoding("my content encoding")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).contentEncoding().get()) + .isEqualTo("my content encoding"); + } + + @Test + public void testUserMetadata() throws IOException { + Files.write(path, HAPPY, withUserMetadata("green", "bean")); + assertThat( + Files.readAttributes(path, CloudStorageFileAttributes.class) + .userMetadata() + .get("green")) + .isEqualTo("bean"); + } + + @Test + public void testIsDirectory() throws IOException { + Files.write(path, HAPPY); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).isDirectory()) + .isFalse(); + assertThat(Files.readAttributes(dir, CloudStorageFileAttributes.class).isDirectory()).isTrue(); + } + + @Test + public void testIsRegularFile() throws IOException { + Files.write(path, HAPPY); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).isRegularFile()) + .isTrue(); + assertThat(Files.readAttributes(dir, CloudStorageFileAttributes.class).isRegularFile()) + .isFalse(); + } + + @Test + public void testIsOther() throws IOException { + Files.write(path, HAPPY); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).isOther()).isFalse(); + assertThat(Files.readAttributes(dir, CloudStorageFileAttributes.class).isOther()).isFalse(); + } + + @Test + public void testIsSymbolicLink() throws IOException { + Files.write(path, HAPPY); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).isSymbolicLink()) + .isFalse(); + assertThat(Files.readAttributes(dir, CloudStorageFileAttributes.class).isSymbolicLink()) + .isFalse(); + } + + @Test + public void testEquals_equalsTester() throws IOException { + Files.write(path, HAPPY, withMimeType("text/plain")); + CloudStorageFileAttributes a1 = Files.readAttributes(path, CloudStorageFileAttributes.class); + CloudStorageFileAttributes a2 = Files.readAttributes(path, CloudStorageFileAttributes.class); + Files.write(path, HAPPY, withMimeType("text/potato")); + CloudStorageFileAttributes b1 = Files.readAttributes(path, CloudStorageFileAttributes.class); + CloudStorageFileAttributes b2 = Files.readAttributes(path, CloudStorageFileAttributes.class); + new EqualsTester().addEqualityGroup(a1, a2).addEqualityGroup(b1, b2).testEquals(); + } + + @Test + public void testFilekey() throws IOException { + Files.write(path, HAPPY, withMimeType("text/plain")); + Path path2 = Paths.get(URI.create("gs://bucket/anotherrandompath")); + Files.write(path2, HAPPY, withMimeType("text/plain")); + + // diff files cannot have same filekey + CloudStorageFileAttributes a1 = Files.readAttributes(path, CloudStorageFileAttributes.class); + CloudStorageFileAttributes a2 = Files.readAttributes(path2, CloudStorageFileAttributes.class); + assertThat(a1.fileKey()).isNotEqualTo(a2.fileKey()); + + // same for directories + CloudStorageFileAttributes b1 = Files.readAttributes(dir, CloudStorageFileAttributes.class); + CloudStorageFileAttributes b2 = + Files.readAttributes( + Paths.get(URI.create("gs://bucket/jacket/")), CloudStorageFileAttributes.class); + assertThat(a1.fileKey()).isNotEqualTo(b1.fileKey()); + assertThat(b1.fileKey()).isNotEqualTo(b2.fileKey()); + } + + @Test + public void testNullness() throws IOException, NoSuchMethodException, SecurityException { + Files.write(path, HAPPY); + CloudStorageFileAttributes pathAttributes = + Files.readAttributes(path, CloudStorageFileAttributes.class); + CloudStorageFileAttributes dirAttributes = + Files.readAttributes(dir, CloudStorageFileAttributes.class); + NullPointerTester tester = new NullPointerTester(); + tester.ignore(CloudStorageObjectAttributes.class.getMethod("equals", Object.class)); + tester.testAllPublicInstanceMethods(pathAttributes); + tester.testAllPublicInstanceMethods(dirAttributes); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProviderTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProviderTest.java new file mode 100644 index 000000000000..f38860aef1e6 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemProviderTest.java @@ -0,0 +1,645 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.forBucket; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withCacheControl; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentDisposition; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentEncoding; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withMimeType; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withUserMetadata; +import static java.nio.charset.StandardCharsets.UTF_8; +import static java.nio.file.StandardCopyOption.ATOMIC_MOVE; +import static java.nio.file.StandardCopyOption.COPY_ATTRIBUTES; +import static java.nio.file.StandardCopyOption.REPLACE_EXISTING; +import static java.nio.file.StandardOpenOption.CREATE_NEW; +import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING; +import static java.nio.file.StandardOpenOption.WRITE; + +import com.google.common.collect.ImmutableList; +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.URI; +import java.nio.ByteBuffer; +import java.nio.channels.ReadableByteChannel; +import java.nio.channels.SeekableByteChannel; +import java.nio.file.AtomicMoveNotSupportedException; +import java.nio.file.CopyOption; +import java.nio.file.FileAlreadyExistsException; +import java.nio.file.FileSystem; +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.NoSuchFileException; +import java.nio.file.OpenOption; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.nio.file.StandardOpenOption; +import java.util.List; + +/** + * Unit tests for {@link CloudStorageFileSystemProvider}. + */ +@RunWith(JUnit4.class) +public class CloudStorageFileSystemProviderTest { + + private static final List FILE_CONTENTS = + ImmutableList.of( + "Fanatics have their dreams, wherewith they weave", + "A paradise for a sect; the savage too", + "From forth the loftiest fashion of his sleep", + "Guesses at Heaven; pity these have not", + "Trac'd upon vellum or wild Indian leaf", + "The shadows of melodious utterance.", + "But bare of laurel they live, dream, and die;", + "For Poesy alone can tell her dreams,", + "With the fine spell of words alone can save", + "Imagination from the sable charm", + "And dumb enchantment. Who alive can say,", + "'Thou art no Poet may'st not tell thy dreams?'", + "Since every man whose soul is not a clod", + "Hath visions, and would speak, if he had loved", + "And been well nurtured in his mother tongue.", + "Whether the dream now purpos'd to rehearse", + "Be poet's or fanatic's will be known", + "When this warm scribe my hand is in the grave."); + + private static final String SINGULARITY = "A string"; + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + } + + @Test + public void testSize() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + assertThat(Files.size(path)).isEqualTo(SINGULARITY.getBytes(UTF_8).length); + } + + @Test + public void testSize_trailingSlash_returnsFakePseudoDirectorySize() throws IOException { + assertThat(Files.size(Paths.get(URI.create("gs://bucket/wat/")))).isEqualTo(1); + } + + @Test + public void testSize_trailingSlash_disablePseudoDirectories() throws IOException { + try (CloudStorageFileSystem fs = forBucket("doodle", usePseudoDirectories(false))) { + Path path = fs.getPath("wat/"); + byte[] rapture = SINGULARITY.getBytes(UTF_8); + Files.write(path, rapture); + assertThat(Files.size(path)).isEqualTo(rapture.length); + } + } + + @Test + public void testReadAllBytes() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + assertThat(new String(Files.readAllBytes(path), UTF_8)).isEqualTo(SINGULARITY); + } + + @Test + public void testReadAllBytes_trailingSlash() throws IOException { + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.readAllBytes(Paths.get(URI.create("gs://bucket/wat/"))); + } + + @Test + public void testNewByteChannelRead() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + byte[] data = SINGULARITY.getBytes(UTF_8); + Files.write(path, data); + try (ReadableByteChannel input = Files.newByteChannel(path)) { + ByteBuffer buffer = ByteBuffer.allocate(data.length); + assertThat(input.read(buffer)).isEqualTo(data.length); + assertThat(new String(buffer.array(), UTF_8)).isEqualTo(SINGULARITY); + buffer.rewind(); + assertThat(input.read(buffer)).isEqualTo(-1); + } + } + + @Test + public void testNewByteChannelRead_seeking() throws IOException { + Path path = Paths.get(URI.create("gs://lol/cat")); + Files.write(path, "helloworld".getBytes(UTF_8)); + try (SeekableByteChannel input = Files.newByteChannel(path)) { + ByteBuffer buffer = ByteBuffer.allocate(5); + input.position(5); + assertThat(input.position()).isEqualTo(5); + assertThat(input.read(buffer)).isEqualTo(5); + assertThat(input.position()).isEqualTo(10); + assertThat(new String(buffer.array(), UTF_8)).isEqualTo("world"); + buffer.rewind(); + assertThat(input.read(buffer)).isEqualTo(-1); + input.position(0); + assertThat(input.position()).isEqualTo(0); + assertThat(input.read(buffer)).isEqualTo(5); + assertThat(input.position()).isEqualTo(5); + assertThat(new String(buffer.array(), UTF_8)).isEqualTo("hello"); + } + } + + @Test + public void testNewByteChannelRead_seekBeyondSize_reportsEofOnNextRead() throws IOException { + Path path = Paths.get(URI.create("gs://lol/cat")); + Files.write(path, "hellocat".getBytes(UTF_8)); + try (SeekableByteChannel input = Files.newByteChannel(path)) { + ByteBuffer buffer = ByteBuffer.allocate(5); + input.position(10); + assertThat(input.read(buffer)).isEqualTo(-1); + input.position(11); + assertThat(input.read(buffer)).isEqualTo(-1); + assertThat(input.size()).isEqualTo(8); + } + } + + @Test + public void testNewByteChannelRead_trailingSlash() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat/")); + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.newByteChannel(path); + } + + @Test + public void testNewByteChannelRead_notFound() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wednesday")); + thrown.expect(NoSuchFileException.class); + Files.newByteChannel(path); + } + + @Test + public void testNewByteChannelWrite() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/tests")); + try (SeekableByteChannel output = Files.newByteChannel(path, WRITE)) { + assertThat(output.position()).isEqualTo(0); + assertThat(output.size()).isEqualTo(0); + ByteBuffer buffer = ByteBuffer.wrap("filec".getBytes(UTF_8)); + assertThat(output.write(buffer)).isEqualTo(5); + assertThat(output.position()).isEqualTo(5); + assertThat(output.size()).isEqualTo(5); + buffer = ByteBuffer.wrap("onten".getBytes(UTF_8)); + assertThat(output.write(buffer)).isEqualTo(5); + assertThat(output.position()).isEqualTo(10); + assertThat(output.size()).isEqualTo(10); + } + assertThat(new String(Files.readAllBytes(path), UTF_8)).isEqualTo("fileconten"); + } + + @Test + public void testNewInputStream() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + try (InputStream input = Files.newInputStream(path)) { + byte[] data = new byte[SINGULARITY.getBytes(UTF_8).length]; + input.read(data); + assertThat(new String(data, UTF_8)).isEqualTo(SINGULARITY); + } + } + + @Test + public void testNewInputStream_trailingSlash() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat/")); + thrown.expect(CloudStoragePseudoDirectoryException.class); + try (InputStream input = Files.newInputStream(path)) { + input.read(); + } + } + + @Test + public void testNewInputStream_notFound() throws IOException { + Path path = Paths.get(URI.create("gs://cry/wednesday")); + thrown.expect(NoSuchFileException.class); + try (InputStream input = Files.newInputStream(path)) { + input.read(); + } + } + + @Test + public void testNewOutputStream() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + try (OutputStream output = Files.newOutputStream(path)) { + output.write(SINGULARITY.getBytes(UTF_8)); + } + assertThat(new String(Files.readAllBytes(path), UTF_8)).isEqualTo(SINGULARITY); + } + + @Test + public void testNewOutputStream_truncateByDefault() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + Files.write(path, "hello".getBytes(UTF_8)); + try (OutputStream output = Files.newOutputStream(path)) { + output.write(SINGULARITY.getBytes(UTF_8)); + } + assertThat(new String(Files.readAllBytes(path), UTF_8)).isEqualTo(SINGULARITY); + } + + @Test + public void testNewOutputStream_truncateExplicitly() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + Files.write(path, "hello".getBytes(UTF_8)); + try (OutputStream output = Files.newOutputStream(path, TRUNCATE_EXISTING)) { + output.write(SINGULARITY.getBytes(UTF_8)); + } + assertThat(new String(Files.readAllBytes(path), UTF_8)).isEqualTo(SINGULARITY); + } + + @Test + public void testNewOutputStream_trailingSlash() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/wat/")); + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.newOutputStream(path); + } + + @Test + public void testNewOutputStream_createNew() throws IOException { + Path path = Paths.get(URI.create("gs://cry/wednesday")); + Files.newOutputStream(path, CREATE_NEW); + } + + @Test + public void testNewOutputStream_createNew_alreadyExists() throws IOException { + Path path = Paths.get(URI.create("gs://cry/wednesday")); + Files.write(path, SINGULARITY.getBytes(UTF_8)); + thrown.expect(FileAlreadyExistsException.class); + Files.newOutputStream(path, CREATE_NEW); + } + + @Test + public void testWrite_objectNameWithExtraSlashes_throwsIae() throws IOException { + Path path = Paths.get(URI.create("gs://double/slash//yep")); + thrown.expect(IllegalArgumentException.class); + Files.write(path, FILE_CONTENTS, UTF_8); + } + + @Test + public void testWrite_objectNameWithExtraSlashes_canBeNormalized() throws IOException { + try (CloudStorageFileSystem fs = forBucket("greenbean", permitEmptyPathComponents(false))) { + Path path = fs.getPath("adipose//yep").normalize(); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.readAllLines(path, UTF_8)).isEqualTo(FILE_CONTENTS); + assertThat(Files.exists(fs.getPath("adipose", "yep"))).isTrue(); + } + } + + @Test + public void testWrite_objectNameWithExtraSlashes_permitEmptyPathComponents() throws IOException { + try (CloudStorageFileSystem fs = forBucket("greenbean", permitEmptyPathComponents(true))) { + Path path = fs.getPath("adipose//yep"); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.readAllLines(path, UTF_8)).isEqualTo(FILE_CONTENTS); + assertThat(Files.exists(path)).isTrue(); + } + } + + @Test + public void testWrite_absoluteObjectName_prefixSlashGetsRemoved() throws IOException { + Path path = Paths.get(URI.create("gs://greenbean/adipose/yep")); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.readAllLines(path, UTF_8)).isEqualTo(FILE_CONTENTS); + assertThat(Files.exists(path)).isTrue(); + } + + @Test + public void testWrite_absoluteObjectName_disableStrip_slashGetsPreserved() throws IOException { + try (CloudStorageFileSystem fs = + forBucket( + "greenbean", CloudStorageConfiguration.builder().stripPrefixSlash(false).build())) { + Path path = fs.getPath("/adipose/yep"); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.readAllLines(path, UTF_8)).isEqualTo(FILE_CONTENTS); + assertThat(Files.exists(path)).isTrue(); + } + } + + @Test + public void testWrite() throws IOException { + Path path = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.readAllLines(path, UTF_8)).isEqualTo(FILE_CONTENTS); + } + + @Test + public void testWriteOnClose() throws IOException { + Path path = Paths.get(URI.create("gs://greenbean/adipose")); + try (SeekableByteChannel chan = Files.newByteChannel(path, StandardOpenOption.WRITE)) { + // writing lots of contents to defeat channel-internal buffering. + for (int i = 0; i < 9999; i++) { + for (String s : FILE_CONTENTS) { + chan.write(ByteBuffer.wrap(s.getBytes(UTF_8))); + } + } + try { + Files.size(path); + // we shouldn't make it to this line. Not using thrown.expect because + // I still want to run a few lines after the exception. + assertThat(false).isTrue(); + } catch (NoSuchFileException nsf) { + // that's what we wanted, we're good. + } + } + // channel now closed, the file should be there and with the new contents. + assertThat(Files.exists(path)).isTrue(); + assertThat(Files.size(path)).isGreaterThan(100L); + } + + @Test + public void testWrite_trailingSlash() throws IOException { + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.write(Paths.get(URI.create("gs://greenbean/adipose/")), FILE_CONTENTS, UTF_8); + } + + @Test + public void testExists() throws IOException { + assertThat(Files.exists(Paths.get(URI.create("gs://military/fashion")))).isFalse(); + Files.write(Paths.get(URI.create("gs://military/fashion")), "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + assertThat(Files.exists(Paths.get(URI.create("gs://military/fashion")))).isTrue(); + } + + @Test + public void testExists_trailingSlash() { + assertThat(Files.exists(Paths.get(URI.create("gs://military/fashion/")))).isTrue(); + assertThat(Files.exists(Paths.get(URI.create("gs://military/fashion/.")))).isTrue(); + assertThat(Files.exists(Paths.get(URI.create("gs://military/fashion/..")))).isTrue(); + } + + @Test + public void testExists_trailingSlash_disablePseudoDirectories() { + try (CloudStorageFileSystem fs = forBucket("military", usePseudoDirectories(false))) { + assertThat(Files.exists(fs.getPath("fashion/"))).isFalse(); + } + } + + @Test + public void testDelete() throws IOException { + Files.write(Paths.get(URI.create("gs://love/fashion")), "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + assertThat(Files.exists(Paths.get(URI.create("gs://love/fashion")))).isTrue(); + Files.delete(Paths.get(URI.create("gs://love/fashion"))); + assertThat(Files.exists(Paths.get(URI.create("gs://love/fashion")))).isFalse(); + } + + @Test + public void testDelete_dotDirNotNormalized_throwsIae() throws IOException { + thrown.expect(IllegalArgumentException.class); + Files.delete(Paths.get(URI.create("gs://love/fly/../passion"))); + } + + @Test + public void testDelete_trailingSlash() throws IOException { + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.delete(Paths.get(URI.create("gs://love/passion/"))); + } + + @Test + public void testDelete_trailingSlash_disablePseudoDirectories() throws IOException { + try (CloudStorageFileSystem fs = forBucket("pumpkin", usePseudoDirectories(false))) { + Path path = fs.getPath("wat/"); + Files.write(path, FILE_CONTENTS, UTF_8); + assertThat(Files.exists(path)); + Files.delete(path); + assertThat(!Files.exists(path)); + } + } + + @Test + public void testDelete_notFound() throws IOException { + thrown.expect(NoSuchFileException.class); + Files.delete(Paths.get(URI.create("gs://loveh/passionehu"))); + } + + @Test + public void testDeleteIfExists() throws IOException { + Files.write(Paths.get(URI.create("gs://love/passionz")), "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + assertThat(Files.deleteIfExists(Paths.get(URI.create("gs://love/passionz")))).isTrue(); + } + + @Test + public void testDeleteIfExists_trailingSlash() throws IOException { + thrown.expect(CloudStoragePseudoDirectoryException.class); + Files.deleteIfExists(Paths.get(URI.create("gs://love/passion/"))); + } + + @Test + public void testCopy() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + Files.copy(source, target); + assertThat(new String(Files.readAllBytes(target), UTF_8)).isEqualTo("(✿◕ ‿◕ )ノ"); + assertThat(Files.exists(source)).isTrue(); + assertThat(Files.exists(target)).isTrue(); + } + + @Test + public void testCopy_sourceMissing_throwsNoSuchFileException() throws IOException { + thrown.expect(NoSuchFileException.class); + Files.copy( + Paths.get(URI.create("gs://military/fashion.show")), + Paths.get(URI.create("gs://greenbean/adipose"))); + } + + @Test + public void testCopy_targetExists_throwsFileAlreadyExistsException() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + Files.write(target, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + thrown.expect(FileAlreadyExistsException.class); + Files.copy(source, target); + } + + @Test + public void testCopyReplace_targetExists_works() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + Files.write(target, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + Files.copy(source, target, REPLACE_EXISTING); + } + + @Test + public void testCopy_directory_doesNothing() throws IOException { + Path source = Paths.get(URI.create("gs://military/fundir/")); + Path target = Paths.get(URI.create("gs://greenbean/loldir/")); + Files.copy(source, target); + } + + @Test + public void testCopy_atomic_throwsUnsupported() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + thrown.expect(UnsupportedOperationException.class); + Files.copy(source, target, ATOMIC_MOVE); + } + + @Test + public void testMove() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + Files.move(source, target); + assertThat(new String(Files.readAllBytes(target), UTF_8)).isEqualTo("(✿◕ ‿◕ )ノ"); + assertThat(Files.exists(source)).isFalse(); + assertThat(Files.exists(target)).isTrue(); + } + + @Test + public void testCreateDirectory() throws IOException { + Path path = Paths.get(URI.create("gs://greenbean/dir/")); + Files.createDirectory(path); + assertThat(Files.exists(path)).isTrue(); + } + + @Test + public void testMove_atomicMove_notSupported() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write(source, "(✿◕ ‿◕ )ノ".getBytes(UTF_8)); + thrown.expect(AtomicMoveNotSupportedException.class); + Files.move(source, target, ATOMIC_MOVE); + } + + @Test + public void testIsDirectory() throws IOException { + try (FileSystem fs = FileSystems.getFileSystem(URI.create("gs://doodle"))) { + assertThat(Files.isDirectory(fs.getPath(""))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("/"))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("."))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("./"))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("cat/.."))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("hello/cat/.."))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("cat/../"))).isTrue(); + assertThat(Files.isDirectory(fs.getPath("hello/cat/../"))).isTrue(); + } + } + + @Test + public void testIsDirectory_trailingSlash_alwaysTrue() { + assertThat(Files.isDirectory(Paths.get(URI.create("gs://military/fundir/")))).isTrue(); + } + + @Test + public void testIsDirectory_trailingSlash_pseudoDirectoriesDisabled_false() { + try (CloudStorageFileSystem fs = forBucket("doodle", usePseudoDirectories(false))) { + assertThat(Files.isDirectory(fs.getPath("fundir/"))).isFalse(); + } + } + + @Test + public void testCopy_withCopyAttributes_preservesAttributes() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write( + source, + "(✿◕ ‿◕ )ノ".getBytes(UTF_8), + withMimeType("text/lolcat"), + withCacheControl("public; max-age=666"), + withContentEncoding("foobar"), + withContentDisposition("my-content-disposition"), + withUserMetadata("answer", "42")); + Files.copy(source, target, COPY_ATTRIBUTES); + + CloudStorageFileAttributes attributes = + Files.readAttributes(target, CloudStorageFileAttributes.class); + assertThat(attributes.mimeType()).hasValue("text/lolcat"); + assertThat(attributes.cacheControl()).hasValue("public; max-age=666"); + assertThat(attributes.contentEncoding()).hasValue("foobar"); + assertThat(attributes.contentDisposition()).hasValue("my-content-disposition"); + assertThat(attributes.userMetadata().containsKey("answer")).isTrue(); + assertThat(attributes.userMetadata().get("answer")).isEqualTo("42"); + } + + @Test + public void testCopy_withoutOptions_doesntPreservesAttributes() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target = Paths.get(URI.create("gs://greenbean/adipose")); + Files.write( + source, + "(✿◕ ‿◕ )ノ".getBytes(UTF_8), + withMimeType("text/lolcat"), + withCacheControl("public; max-age=666"), + withUserMetadata("answer", "42")); + Files.copy(source, target); + + CloudStorageFileAttributes attributes = + Files.readAttributes(target, CloudStorageFileAttributes.class); + String mimeType = attributes.mimeType().orNull(); + String cacheControl = attributes.cacheControl().orNull(); + assertThat(mimeType).isNotEqualTo("text/lolcat"); + assertThat(cacheControl).isNull(); + assertThat(attributes.userMetadata().containsKey("answer")).isFalse(); + } + + @Test + public void testCopy_overwriteAttributes() throws IOException { + Path source = Paths.get(URI.create("gs://military/fashion.show")); + Path target1 = Paths.get(URI.create("gs://greenbean/adipose")); + Path target2 = Paths.get(URI.create("gs://greenbean/round")); + Files.write( + source, + "(✿◕ ‿◕ )ノ".getBytes(UTF_8), + withMimeType("text/lolcat"), + withCacheControl("public; max-age=666")); + Files.copy(source, target1, COPY_ATTRIBUTES); + Files.copy(source, target2, COPY_ATTRIBUTES, withMimeType("text/palfun")); + + CloudStorageFileAttributes attributes = + Files.readAttributes(target1, CloudStorageFileAttributes.class); + assertThat(attributes.mimeType()).hasValue("text/lolcat"); + assertThat(attributes.cacheControl()).hasValue("public; max-age=666"); + + attributes = Files.readAttributes(target2, CloudStorageFileAttributes.class); + assertThat(attributes.mimeType()).hasValue("text/palfun"); + assertThat(attributes.cacheControl()).hasValue("public; max-age=666"); + } + + @Test + public void testNullness() throws IOException, NoSuchMethodException, SecurityException { + try (FileSystem fs = FileSystems.getFileSystem(URI.create("gs://blood"))) { + NullPointerTester tester = new NullPointerTester(); + tester.ignore(CloudStorageFileSystemProvider.class.getMethod("equals", Object.class)); + tester.setDefault(URI.class, URI.create("gs://blood")); + tester.setDefault(Path.class, fs.getPath("and/one")); + tester.setDefault(OpenOption.class, StandardOpenOption.CREATE); + tester.setDefault(CopyOption.class, StandardCopyOption.COPY_ATTRIBUTES); + // can't do that, setGCloudOptions accepts a null argument. + // TODO(jart): Figure out how to re-enable this. + // tester.testAllPublicStaticMethods(CloudStorageFileSystemProvider.class); + tester.testAllPublicInstanceMethods(new CloudStorageFileSystemProvider()); + } + } + + @Test + public void testProviderEquals() { + Path path1 = Paths.get(URI.create("gs://bucket/tuesday")); + Path path2 = Paths.get(URI.create("gs://blood/wednesday")); + Path path3 = Paths.get("tmp"); + assertThat(path1.getFileSystem().provider()).isEqualTo(path2.getFileSystem().provider()); + assertThat(path1.getFileSystem().provider()).isNotEqualTo(path3.getFileSystem().provider()); + } + + private static CloudStorageConfiguration permitEmptyPathComponents(boolean value) { + return CloudStorageConfiguration.builder().permitEmptyPathComponents(value).build(); + } + + private static CloudStorageConfiguration usePseudoDirectories(boolean value) { + return CloudStorageConfiguration.builder().usePseudoDirectories(value).build(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemTest.java new file mode 100644 index 000000000000..3ef113e14744 --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageFileSystemTest.java @@ -0,0 +1,119 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.common.testing.EqualsTester; +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.FileSystem; +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.Paths; + +/** + * Unit tests for {@link CloudStorageFileSystem}. + */ +@RunWith(JUnit4.class) +public class CloudStorageFileSystemTest { + + private static final String ALONE = + "To be, or not to be, that is the question—\n" + + "Whether 'tis Nobler in the mind to suffer\n" + + "The Slings and Arrows of outrageous Fortune,\n" + + "Or to take Arms against a Sea of troubles,\n" + + "And by opposing, end them? To die, to sleep—\n" + + "No more; and by a sleep, to say we end\n" + + "The Heart-ache, and the thousand Natural shocks\n" + + "That Flesh is heir to? 'Tis a consummation\n"; + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + } + + @Test + public void testGetPath() throws IOException { + try (FileSystem fs = CloudStorageFileSystem.forBucket("bucket")) { + assertThat(fs.getPath("/angel").toString()).isEqualTo("/angel"); + assertThat(fs.getPath("/angel").toUri().toString()).isEqualTo("gs://bucket/angel"); + } + } + + @Test + public void testWrite() throws IOException { + try (FileSystem fs = CloudStorageFileSystem.forBucket("bucket")) { + Files.write(fs.getPath("/angel"), ALONE.getBytes(UTF_8)); + } + assertThat(new String(Files.readAllBytes(Paths.get(URI.create("gs://bucket/angel"))), UTF_8)) + .isEqualTo(ALONE); + } + + @Test + public void testRead() throws IOException { + Files.write(Paths.get(URI.create("gs://bucket/angel")), ALONE.getBytes(UTF_8)); + try (FileSystem fs = CloudStorageFileSystem.forBucket("bucket")) { + assertThat(new String(Files.readAllBytes(fs.getPath("/angel")), UTF_8)).isEqualTo(ALONE); + } + } + + @Test + public void testExists_false() throws IOException { + try (FileSystem fs = FileSystems.getFileSystem(URI.create("gs://bucket"))) { + assertThat(Files.exists(fs.getPath("/angel"))).isFalse(); + } + } + + @Test + public void testExists_true() throws IOException { + Files.write(Paths.get(URI.create("gs://bucket/angel")), ALONE.getBytes(UTF_8)); + try (FileSystem fs = CloudStorageFileSystem.forBucket("bucket")) { + assertThat(Files.exists(fs.getPath("/angel"))).isTrue(); + } + } + + @Test + public void testGetters() throws IOException { + try (FileSystem fs = CloudStorageFileSystem.forBucket("bucket")) { + assertThat(fs.isOpen()).isTrue(); + assertThat(fs.isReadOnly()).isFalse(); + assertThat(fs.getRootDirectories()).containsExactly(fs.getPath("/")); + assertThat(fs.getFileStores()).isEmpty(); + assertThat(fs.getSeparator()).isEqualTo("/"); + assertThat(fs.supportedFileAttributeViews()).containsExactly("basic", "gcs"); + } + } + + @Test + public void testEquals() throws IOException { + try (FileSystem bucket1 = CloudStorageFileSystem.forBucket("bucket"); + FileSystem bucket2 = FileSystems.getFileSystem(URI.create("gs://bucket")); + FileSystem doge1 = CloudStorageFileSystem.forBucket("doge"); + FileSystem doge2 = FileSystems.getFileSystem(URI.create("gs://doge"))) { + new EqualsTester() + .addEqualityGroup(bucket1, bucket2) + .addEqualityGroup(doge1, doge2) + .testEquals(); + } + } + + @Test + public void testNullness() throws IOException, NoSuchMethodException, SecurityException { + try (FileSystem fs = FileSystems.getFileSystem(URI.create("gs://bucket"))) { + NullPointerTester tester = + new NullPointerTester() + .ignore(CloudStorageFileSystem.class.getMethod("equals", Object.class)) + .setDefault(CloudStorageConfiguration.class, CloudStorageConfiguration.DEFAULT); + tester.testAllPublicStaticMethods(CloudStorageFileSystem.class); + tester.testAllPublicInstanceMethods(fs); + } + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptionsTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptionsTest.java new file mode 100644 index 000000000000..e852d4035a1a --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageOptionsTest.java @@ -0,0 +1,112 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withAcl; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withCacheControl; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentDisposition; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withContentEncoding; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withMimeType; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withUserMetadata; +import static com.google.gcloud.storage.contrib.nio.CloudStorageOptions.withoutCaching; +import static java.nio.charset.StandardCharsets.UTF_8; + +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.Acl; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** + * Unit tests for {@link CloudStorageOptions}. + */ +@RunWith(JUnit4.class) +public class CloudStorageOptionsTest { + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + } + + @Test + public void testWithoutCaching() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withoutCaching()); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).cacheControl().get()) + .isEqualTo("no-cache"); + } + + @Test + public void testCacheControl() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withCacheControl("potato")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).cacheControl().get()) + .isEqualTo("potato"); + } + + @Test + public void testWithAcl() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Acl acl = Acl.of(new Acl.User("king@example.com"), Acl.Role.OWNER); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withAcl(acl)); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).acl().get()) + .contains(acl); + } + + @Test + public void testWithContentDisposition() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withContentDisposition("bubbly fun")); + assertThat( + Files.readAttributes(path, CloudStorageFileAttributes.class).contentDisposition().get()) + .isEqualTo("bubbly fun"); + } + + @Test + public void testWithContentEncoding() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withContentEncoding("gzip")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).contentEncoding().get()) + .isEqualTo("gzip"); + } + + @Test + public void testWithUserMetadata() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write( + path, + "(✿◕ ‿◕ )ノ".getBytes(UTF_8), + withUserMetadata("nolo", "contendere"), + withUserMetadata("eternal", "sadness")); + assertThat( + Files.readAttributes(path, CloudStorageFileAttributes.class).userMetadata().get("nolo")) + .isEqualTo("contendere"); + assertThat( + Files.readAttributes(path, CloudStorageFileAttributes.class) + .userMetadata() + .get("eternal")) + .isEqualTo("sadness"); + } + + @Test + public void testWithMimeType_string() throws IOException { + Path path = Paths.get(URI.create("gs://bucket/path")); + Files.write(path, "(✿◕ ‿◕ )ノ".getBytes(UTF_8), withMimeType("text/plain")); + assertThat(Files.readAttributes(path, CloudStorageFileAttributes.class).mimeType().get()) + .isEqualTo("text/plain"); + } + + @Test + public void testNullness() { + NullPointerTester tester = new NullPointerTester(); + tester.testAllPublicStaticMethods(CloudStorageOptions.class); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStoragePathTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStoragePathTest.java new file mode 100644 index 000000000000..24393f175b2b --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStoragePathTest.java @@ -0,0 +1,484 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static com.google.gcloud.storage.contrib.nio.CloudStorageFileSystem.forBucket; + +import com.google.common.collect.Iterables; +import com.google.common.testing.EqualsTester; +import com.google.common.testing.NullPointerTester; +import com.google.gcloud.storage.testing.LocalGcsHelper; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.net.URI; +import java.nio.file.FileSystem; +import java.nio.file.FileSystems; +import java.nio.file.Path; +import java.nio.file.ProviderMismatchException; + +/** + * Unit tests for {@link CloudStoragePath}. + */ +@RunWith(JUnit4.class) +public class CloudStoragePathTest { + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + @Before + public void before() { + CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options()); + } + + @Test + public void testCreate_neverRemoveExtraSlashes() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("lol//cat").toString()).isEqualTo("lol//cat"); + assertThat((Object) fs.getPath("lol//cat")).isEqualTo(fs.getPath("lol//cat")); + } + } + + @Test + public void testCreate_preservesTrailingSlash() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("lol/cat/").toString()).isEqualTo("lol/cat/"); + assertThat((Object) fs.getPath("lol/cat/")).isEqualTo(fs.getPath("lol/cat/")); + } + } + + @Test + public void testGetGcsFilename_empty_notAllowed() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + fs.getPath("").getBlobId(); + } + } + + @Test + public void testGetGcsFilename_stripsPrefixSlash() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi").getBlobId().name()).isEqualTo("hi"); + } + } + + @Test + public void testGetGcsFilename_overrideStripPrefixSlash_doesntStripPrefixSlash() { + try (CloudStorageFileSystem fs = forBucket("doodle", stripPrefixSlash(false))) { + assertThat(fs.getPath("/hi").getBlobId().name()).isEqualTo("/hi"); + } + } + + @Test + public void testGetGcsFilename_extraSlashes_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + fs.getPath("a//b").getBlobId().name(); + } + } + + @Test + public void testGetGcsFilename_overridepermitEmptyPathComponents() { + try (CloudStorageFileSystem fs = forBucket("doodle", permitEmptyPathComponents(true))) { + assertThat(fs.getPath("a//b").getBlobId().name()).isEqualTo("a//b"); + } + } + + @Test + public void testGetGcsFilename_freaksOutOnExtraSlashesAndDotDirs() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + fs.getPath("a//b/..").getBlobId().name(); + } + } + + @Test + public void testNameCount() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("").getNameCount()).isEqualTo(1); + assertThat(fs.getPath("/").getNameCount()).isEqualTo(0); + assertThat(fs.getPath("/hi/").getNameCount()).isEqualTo(1); + assertThat(fs.getPath("/hi/yo").getNameCount()).isEqualTo(2); + assertThat(fs.getPath("hi/yo").getNameCount()).isEqualTo(2); + } + } + + @Test + public void testGetName() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("").getName(0).toString()).isEqualTo(""); + assertThat(fs.getPath("/hi").getName(0).toString()).isEqualTo("hi"); + assertThat(fs.getPath("hi/there").getName(1).toString()).isEqualTo("there"); + } + } + + @Test + public void testGetName_negative_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + fs.getPath("angel").getName(-1); + } + } + + @Test + public void testGetName_overflow_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + fs.getPath("angel").getName(1); + } + } + + @Test + public void testIterator() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(Iterables.get(fs.getPath("/dog/mog"), 0).toString()).isEqualTo("dog"); + assertThat(Iterables.get(fs.getPath("/dog/mog"), 1).toString()).isEqualTo("mog"); + assertThat(Iterables.size(fs.getPath("/"))).isEqualTo(0); + assertThat(Iterables.size(fs.getPath(""))).isEqualTo(1); + assertThat(Iterables.get(fs.getPath(""), 0).toString()).isEqualTo(""); + } + } + + @Test + public void testNormalize() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/").normalize().toString()).isEqualTo("/"); + assertThat(fs.getPath("a/x/../b/x/..").normalize().toString()).isEqualTo("a/b/"); + assertThat(fs.getPath("/x/x/../../♡").normalize().toString()).isEqualTo("/♡"); + assertThat(fs.getPath("/x/x/./.././.././♡").normalize().toString()).isEqualTo("/♡"); + } + } + + @Test + public void testNormalize_dot_becomesBlank() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("").normalize().toString()).isEqualTo(""); + assertThat(fs.getPath(".").normalize().toString()).isEqualTo(""); + } + } + + @Test + public void testNormalize_trailingSlash_isPreserved() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("o/").normalize().toString()).isEqualTo("o/"); + } + } + + @Test + public void testNormalize_doubleDot_becomesBlank() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("..").normalize().toString()).isEqualTo(""); + assertThat(fs.getPath("../..").normalize().toString()).isEqualTo(""); + } + } + + @Test + public void testNormalize_extraSlashes_getRemoved() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("//life///b/good//").normalize().toString()).isEqualTo("/life/b/good/"); + } + } + + @Test + public void testToRealPath_hasDotDir_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + fs.getPath("a/hi./b").toRealPath(); + fs.getPath("a/.hi/b").toRealPath(); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage("dot-dir"); + fs.getPath("a/./b").toRealPath(); + } + } + + @Test + public void testToRealPath_hasDotDotDir_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + fs.getPath("a/hi../b").toRealPath(); + fs.getPath("a/..hi/b").toRealPath(); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage("dot-dir"); + fs.getPath("a/../b").toRealPath(); + } + } + + @Test + public void testToRealPath_extraSlashes_throwsIae() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage("extra slashes"); + fs.getPath("a//b").toRealPath(); + } + } + + @Test + public void testToRealPath_overridePermitEmptyPathComponents_extraSlashes_slashesRemain() { + try (CloudStorageFileSystem fs = forBucket("doodle", permitEmptyPathComponents(true))) { + assertThat(fs.getPath("/life///b/./good/").toRealPath().toString()) + .isEqualTo("life///b/./good/"); + } + } + + @Test + public void testToRealPath_permitEmptyPathComponents_doesNotNormalize() { + try (CloudStorageFileSystem fs = forBucket("doodle", permitEmptyPathComponents(true))) { + assertThat(fs.getPath("a").toRealPath().toString()).isEqualTo("a"); + assertThat(fs.getPath("a//b").toRealPath().toString()).isEqualTo("a//b"); + assertThat(fs.getPath("a//./b//..").toRealPath().toString()).isEqualTo("a//./b//.."); + } + } + + @Test + public void testToRealPath_withWorkingDirectory_makesAbsolute() { + try (CloudStorageFileSystem fs = forBucket("doodle", workingDirectory("/lol"))) { + assertThat(fs.getPath("a").toRealPath().toString()).isEqualTo("lol/a"); + } + } + + @Test + public void testToRealPath_disableStripPrefixSlash_makesPathAbsolute() { + try (CloudStorageFileSystem fs = forBucket("doodle", stripPrefixSlash(false))) { + assertThat(fs.getPath("a").toRealPath().toString()).isEqualTo("/a"); + assertThat(fs.getPath("/a").toRealPath().toString()).isEqualTo("/a"); + } + } + + @Test + public void testToRealPath_trailingSlash_getsPreserved() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("a/b/").toRealPath().toString()).isEqualTo("a/b/"); + } + } + + @Test + public void testNormalize_empty_returnsEmpty() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("").normalize().toString()).isEqualTo(""); + } + } + + @Test + public void testNormalize_preserveTrailingSlash() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("a/b/../c/").normalize().toString()).isEqualTo("a/c/"); + assertThat(fs.getPath("a/b/./c/").normalize().toString()).isEqualTo("a/b/c/"); + } + } + + @Test + public void testGetParent_preserveTrailingSlash() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("a/b/c").getParent().toString()).isEqualTo("a/b/"); + assertThat(fs.getPath("a/b/c/").getParent().toString()).isEqualTo("a/b/"); + assertThat((Object) fs.getPath("").getParent()).isNull(); + assertThat((Object) fs.getPath("/").getParent()).isNull(); + assertThat((Object) fs.getPath("aaa").getParent()).isNull(); + assertThat((Object) (fs.getPath("aaa/").getParent())).isNull(); + } + } + + @Test + public void testGetRoot() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hello").getRoot().toString()).isEqualTo("/"); + assertThat((Object) fs.getPath("hello").getRoot()).isNull(); + } + } + + @Test + public void testRelativize() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat( + fs.getPath("/foo/bar/lol/cat").relativize(fs.getPath("/foo/a/b/../../c")).toString()) + .isEqualTo("../../../a/b/../../c"); + } + } + + @Test + public void testRelativize_providerMismatch() { + try (CloudStorageFileSystem gcs = forBucket("doodle")) { + thrown.expect(ProviderMismatchException.class); + gcs.getPath("/etc").relativize(FileSystems.getDefault().getPath("/dog")); + } + } + + @Test + @SuppressWarnings("ReturnValueIgnored") // testing that an Exception is thrown + public void testRelativize_providerMismatch2() { + try (CloudStorageFileSystem gcs = forBucket("doodle")) { + thrown.expect(ProviderMismatchException.class); + gcs.getPath("/dog").relativize(FileSystems.getDefault().getPath("/etc")); + } + } + + @Test + public void testResolve() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi").resolve("there").toString()).isEqualTo("/hi/there"); + assertThat(fs.getPath("hi").resolve("there").toString()).isEqualTo("hi/there"); + } + } + + @Test + public void testResolve_providerMismatch() { + try (CloudStorageFileSystem gcs = forBucket("doodle")) { + thrown.expect(ProviderMismatchException.class); + gcs.getPath("etc").resolve(FileSystems.getDefault().getPath("/dog")); + } + } + + @Test + public void testIsAbsolute() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi").isAbsolute()).isTrue(); + assertThat(fs.getPath("hi").isAbsolute()).isFalse(); + } + } + + @Test + public void testToAbsolutePath() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat((Object) fs.getPath("/hi").toAbsolutePath()).isEqualTo(fs.getPath("/hi")); + assertThat((Object) fs.getPath("hi").toAbsolutePath()).isEqualTo(fs.getPath("/hi")); + } + } + + @Test + public void testToAbsolutePath_withWorkingDirectory() { + try (CloudStorageFileSystem fs = forBucket("doodle", workingDirectory("/lol"))) { + assertThat(fs.getPath("a").toAbsolutePath().toString()).isEqualTo("/lol/a"); + } + } + + @Test + public void testGetFileName() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi/there").getFileName().toString()).isEqualTo("there"); + assertThat(fs.getPath("military/fashion/show").getFileName().toString()).isEqualTo("show"); + } + } + + @Test + public void testCompareTo() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi/there").compareTo(fs.getPath("/hi/there"))).isEqualTo(0); + assertThat(fs.getPath("/hi/there").compareTo(fs.getPath("/hi/therf"))).isEqualTo(-1); + assertThat(fs.getPath("/hi/there").compareTo(fs.getPath("/hi/therd"))).isEqualTo(1); + } + } + + @Test + public void testStartsWith() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("/hi/there"))).isTrue(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("/hi/therf"))).isFalse(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("/hi"))).isTrue(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("/hi/"))).isTrue(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("hi"))).isFalse(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath("/"))).isTrue(); + assertThat(fs.getPath("/hi/there").startsWith(fs.getPath(""))).isFalse(); + } + } + + @Test + public void testEndsWith() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath("there"))).isTrue(); + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath("therf"))).isFalse(); + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath("/blag/therf"))).isFalse(); + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath("/hi/there"))).isTrue(); + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath("/there"))).isFalse(); + assertThat(fs.getPath("/human/that/you/cry").endsWith(fs.getPath("that/you/cry"))).isTrue(); + assertThat(fs.getPath("/human/that/you/cry").endsWith(fs.getPath("that/you/cry/"))).isTrue(); + assertThat(fs.getPath("/hi/there/").endsWith(fs.getPath("/"))).isFalse(); + assertThat(fs.getPath("/hi/there").endsWith(fs.getPath(""))).isFalse(); + assertThat(fs.getPath("").endsWith(fs.getPath(""))).isTrue(); + } + } + + @Test + public void testResolve_willWorkWithRecursiveCopy() throws IOException { + // See: http://stackoverflow.com/a/10068306 + try (FileSystem fsSource = FileSystems.getFileSystem(URI.create("gs://hello")); + FileSystem fsTarget = FileSystems.getFileSystem(URI.create("gs://cat"))) { + Path targetPath = fsTarget.getPath("/some/folder/"); + Path relSrcPath = fsSource.getPath("file.txt"); + assertThat((Object) targetPath.resolve(relSrcPath)) + .isEqualTo(fsTarget.getPath("/some/folder/file.txt")); + } + } + + @Test + public void testRelativize_willWorkWithRecursiveCopy() throws IOException { + // See: http://stackoverflow.com/a/10068306 + try (FileSystem fsSource = FileSystems.getFileSystem(URI.create("gs://hello")); + FileSystem fsTarget = FileSystems.getFileSystem(URI.create("gs://cat"))) { + Path targetPath = fsTarget.getPath("/some/folder/"); + Path sourcePath = fsSource.getPath("/sloth/"); + Path file = fsSource.getPath("/sloth/file.txt"); + assertThat((Object) targetPath.resolve(sourcePath.relativize(file))) + .isEqualTo(fsTarget.getPath("/some/folder/file.txt")); + } + } + + @Test + public void testToFile_unsupported() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + Path path = fs.getPath("/lol"); + thrown.expect(UnsupportedOperationException.class); + path.toFile(); + } + } + + @Test + public void testEquals() { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + new EqualsTester() + // These are obviously equal. + .addEqualityGroup(fs.getPath("/hello/cat"), fs.getPath("/hello/cat")) + // These are equal because equals() runs things through toRealPath() + .addEqualityGroup(fs.getPath("great/commandment"), fs.getPath("/great/commandment")) + .addEqualityGroup(fs.getPath("great/commandment/"), fs.getPath("/great/commandment/")) + // Equals shouldn't do error checking or normalization. + .addEqualityGroup(fs.getPath("foo/../bar"), fs.getPath("foo/../bar")) + .addEqualityGroup(fs.getPath("bar")) + .testEquals(); + } + } + + @Test + public void testEquals_currentDirectoryIsTakenIntoConsideration() { + try (CloudStorageFileSystem fs = forBucket("doodle", workingDirectory("/hello"))) { + new EqualsTester() + .addEqualityGroup(fs.getPath("cat"), fs.getPath("/hello/cat")) + .addEqualityGroup(fs.getPath(""), fs.getPath("/hello")) + .testEquals(); + } + } + + @Test + public void testNullness() throws NoSuchMethodException, SecurityException { + try (CloudStorageFileSystem fs = forBucket("doodle")) { + NullPointerTester tester = new NullPointerTester(); + tester.ignore(CloudStoragePath.class.getMethod("equals", Object.class)); + tester.setDefault(Path.class, fs.getPath("sup")); + tester.testAllPublicStaticMethods(CloudStoragePath.class); + tester.testAllPublicInstanceMethods(fs.getPath("sup")); + } + } + + private static CloudStorageConfiguration stripPrefixSlash(boolean value) { + return CloudStorageConfiguration.builder().stripPrefixSlash(value).build(); + } + + private static CloudStorageConfiguration permitEmptyPathComponents(boolean value) { + return CloudStorageConfiguration.builder().permitEmptyPathComponents(value).build(); + } + + private static CloudStorageConfiguration workingDirectory(String value) { + return CloudStorageConfiguration.builder().workingDirectory(value).build(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannelTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannelTest.java new file mode 100644 index 000000000000..c2c7d50aa68c --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageReadChannelTest.java @@ -0,0 +1,144 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoMoreInteractions; +import static org.mockito.Mockito.verifyZeroInteractions; +import static org.mockito.Mockito.when; + +import com.google.gcloud.ReadChannel; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Storage; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonWritableChannelException; + +/** + * Unit tests for {@link CloudStorageReadChannel}. + */ +@RunWith(JUnit4.class) +public class CloudStorageReadChannelTest { + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + private CloudStorageReadChannel chan; + + private final Storage gcsStorage = mock(Storage.class); + private final BlobId file = BlobId.of("blob", "attack"); + private final BlobInfo metadata = BlobInfo.builder(file).size(42L).build(); + private final ReadChannel gcsChannel = mock(ReadChannel.class); + + @Before + public void before() throws IOException { + when(gcsStorage.get(file)).thenReturn(metadata); + when(gcsStorage.reader(eq(file))).thenReturn(gcsChannel); + when(gcsChannel.isOpen()).thenReturn(true); + chan = CloudStorageReadChannel.create(gcsStorage, file, 0); + verify(gcsStorage).get(eq(file)); + verify(gcsStorage).reader(eq(file)); + } + + @Test + public void testRead() throws IOException { + ByteBuffer buffer = ByteBuffer.allocate(1); + when(gcsChannel.read(eq(buffer))).thenReturn(1); + assertThat(chan.position()).isEqualTo(0L); + assertThat(chan.read(buffer)).isEqualTo(1); + assertThat(chan.position()).isEqualTo(1L); + verify(gcsChannel).read(any(ByteBuffer.class)); + verify(gcsChannel, times(3)).isOpen(); + verifyNoMoreInteractions(gcsStorage, gcsChannel); + } + + @Test + public void testRead_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.read(ByteBuffer.allocate(1)); + } + + @Test + public void testWrite_throwsNonWritableChannelException() throws IOException { + thrown.expect(NonWritableChannelException.class); + chan.write(ByteBuffer.allocate(1)); + } + + @Test + public void testTruncate_throwsNonWritableChannelException() throws IOException { + thrown.expect(NonWritableChannelException.class); + chan.truncate(0); + } + + @Test + public void testIsOpen() throws IOException { + when(gcsChannel.isOpen()).thenReturn(true).thenReturn(false); + assertThat(chan.isOpen()).isTrue(); + chan.close(); + assertThat(chan.isOpen()).isFalse(); + verify(gcsChannel, times(2)).isOpen(); + verify(gcsChannel).close(); + verifyNoMoreInteractions(gcsStorage, gcsChannel); + } + + @Test + public void testSize() throws IOException { + assertThat(chan.size()).isEqualTo(42L); + verify(gcsChannel).isOpen(); + verifyZeroInteractions(gcsChannel); + verifyNoMoreInteractions(gcsStorage); + } + + @Test + public void testSize_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.size(); + } + + @Test + public void testPosition_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.position(); + } + + @Test + public void testSetPosition_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.position(0); + } + + @Test + public void testClose_calledMultipleTimes_doesntThrowAnError() throws IOException { + chan.close(); + chan.close(); + chan.close(); + } + + @Test + public void testSetPosition() throws IOException { + assertThat(chan.position()).isEqualTo(0L); + assertThat(chan.size()).isEqualTo(42L); + chan.position(1L); + assertThat(chan.position()).isEqualTo(1L); + assertThat(chan.size()).isEqualTo(42L); + verify(gcsChannel).seek(1); + verify(gcsChannel, times(5)).isOpen(); + verifyNoMoreInteractions(gcsStorage, gcsChannel); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannelTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannelTest.java new file mode 100644 index 000000000000..60f7d7febd0a --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/CloudStorageWriteChannelTest.java @@ -0,0 +1,109 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoMoreInteractions; +import static org.mockito.Mockito.verifyZeroInteractions; +import static org.mockito.Mockito.when; + +import com.google.gcloud.WriteChannel; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonReadableChannelException; + +/** + * Unit tests for {@link CloudStorageWriteChannel}. + */ +@RunWith(JUnit4.class) +public class CloudStorageWriteChannelTest { + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + private final WriteChannel gcsChannel = mock(WriteChannel.class); + private CloudStorageWriteChannel chan = new CloudStorageWriteChannel(gcsChannel); + + @Before + public void before() { + when(gcsChannel.isOpen()).thenReturn(true); + } + + @Test + public void testRead_throwsNonReadableChannelException() throws IOException { + thrown.expect(NonReadableChannelException.class); + chan.read(ByteBuffer.allocate(1)); + } + + @Test + public void testWrite() throws IOException { + ByteBuffer buffer = ByteBuffer.allocate(1); + buffer.put((byte) 'B'); + assertThat(chan.position()).isEqualTo(0L); + assertThat(chan.size()).isEqualTo(0L); + when(gcsChannel.write(eq(buffer))).thenReturn(1); + assertThat(chan.write(buffer)).isEqualTo(1); + assertThat(chan.position()).isEqualTo(1L); + assertThat(chan.size()).isEqualTo(1L); + verify(gcsChannel).write(any(ByteBuffer.class)); + verify(gcsChannel, times(5)).isOpen(); + verifyNoMoreInteractions(gcsChannel); + } + + @Test + public void testWrite_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.write(ByteBuffer.allocate(1)); + } + + @Test + public void testIsOpen() throws IOException { + when(gcsChannel.isOpen()).thenReturn(true).thenReturn(false); + assertThat(chan.isOpen()).isTrue(); + chan.close(); + assertThat(chan.isOpen()).isFalse(); + verify(gcsChannel, times(2)).isOpen(); + verify(gcsChannel).close(); + verifyNoMoreInteractions(gcsChannel); + } + + @Test + public void testSize() throws IOException { + assertThat(chan.size()).isEqualTo(0L); + verify(gcsChannel).isOpen(); + verifyZeroInteractions(gcsChannel); + } + + @Test + public void testSize_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.size(); + } + + @Test + public void testPosition_whenClosed_throwsCce() throws IOException { + when(gcsChannel.isOpen()).thenReturn(false); + thrown.expect(ClosedChannelException.class); + chan.position(); + } + + @Test + public void testClose_calledMultipleTimes_doesntThrowAnError() throws IOException { + chan.close(); + chan.close(); + chan.close(); + } +} diff --git a/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/UnixPathTest.java b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/UnixPathTest.java new file mode 100644 index 000000000000..4d6ee16a8c7f --- /dev/null +++ b/gcloud-java-contrib/gcloud-java-nio/src/test/java/com/google/gcloud/storage/contrib/nio/UnixPathTest.java @@ -0,0 +1,386 @@ +package com.google.gcloud.storage.contrib.nio; + +import static com.google.common.truth.Truth.assertThat; +import static org.junit.Assume.assumeTrue; + +import com.google.common.testing.EqualsTester; +import com.google.common.testing.NullPointerTester; + +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.junit.runner.RunWith; +import org.junit.runners.JUnit4; + +/** + * Unit tests for {@link UnixPath}. + */ +@RunWith(JUnit4.class) +public class UnixPathTest { + + @Rule public final ExpectedException thrown = ExpectedException.none(); + + @Test + public void testNormalize() { + assertThat(p(".").normalize()).isEqualTo(p("")); + assertThat(p("/").normalize()).isEqualTo(p("/")); + assertThat(p("/.").normalize()).isEqualTo(p("/")); + assertThat(p("/a/b/../c").normalize()).isEqualTo(p("/a/c")); + assertThat(p("/a/b/./c").normalize()).isEqualTo(p("/a/b/c")); + assertThat(p("a/b/../c").normalize()).isEqualTo(p("a/c")); + assertThat(p("a/b/./c").normalize()).isEqualTo(p("a/b/c")); + assertThat(p("/a/b/../../c").normalize()).isEqualTo(p("/c")); + assertThat(p("/a/b/./.././.././c").normalize()).isEqualTo(p("/c")); + } + + @Test + public void testNormalize_empty_returnsEmpty() { + assertThat(p("").normalize()).isEqualTo(p("")); + } + + @Test + public void testNormalize_underflow_isAllowed() { + assertThat(p("../").normalize()).isEqualTo(p("")); + } + + @Test + public void testNormalize_extraSlashes_getRemoved() { + assertThat(p("///").normalize()).isEqualTo(p("/")); + assertThat(p("/hi//there").normalize()).isEqualTo(p("/hi/there")); + assertThat(p("/hi////.///there").normalize()).isEqualTo(p("/hi/there")); + } + + @Test + public void testNormalize_trailingSlash() { + assertThat(p("/hi/there/").normalize()).isEqualTo(p("/hi/there/")); + assertThat(p("/hi/there/../").normalize()).isEqualTo(p("/hi/")); + assertThat(p("/hi/there/..").normalize()).isEqualTo(p("/hi/")); + assertThat(p("hi/../").normalize()).isEqualTo(p("")); + assertThat(p("/hi/../").normalize()).isEqualTo(p("/")); + assertThat(p("hi/..").normalize()).isEqualTo(p("")); + assertThat(p("/hi/..").normalize()).isEqualTo(p("/")); + } + + @Test + public void testNormalize_sameObjectOptimization() { + UnixPath path = p("/hi/there"); + assertThat(path.normalize()).isSameAs(path); + path = p("/hi/there/"); + assertThat(path.normalize()).isSameAs(path); + } + + @Test + public void testResolve() { + assertThat(p("/hello").resolve(p("cat"))).isEqualTo(p("/hello/cat")); + assertThat(p("/hello/").resolve(p("cat"))).isEqualTo(p("/hello/cat")); + assertThat(p("hello/").resolve(p("cat"))).isEqualTo(p("hello/cat")); + assertThat(p("hello/").resolve(p("cat/"))).isEqualTo(p("hello/cat/")); + assertThat(p("hello/").resolve(p(""))).isEqualTo(p("hello/")); + assertThat(p("hello/").resolve(p("/hi/there"))).isEqualTo(p("/hi/there")); + } + + @Test + public void testResolve_sameObjectOptimization() { + UnixPath path = p("/hi/there"); + assertThat(path.resolve(p(""))).isSameAs(path); + assertThat(p("hello").resolve(path)).isSameAs(path); + } + + @Test + public void testGetPath() { + assertThat(UnixPath.getPath(false, "hello")).isEqualTo(p("hello")); + assertThat(UnixPath.getPath(false, "hello", "cat")).isEqualTo(p("hello/cat")); + assertThat(UnixPath.getPath(false, "/hello", "cat")).isEqualTo(p("/hello/cat")); + assertThat(UnixPath.getPath(false, "/hello", "cat", "inc.")).isEqualTo(p("/hello/cat/inc.")); + assertThat(UnixPath.getPath(false, "hello/", "/hi/there")).isEqualTo(p("/hi/there")); + } + + @Test + public void testResolveSibling() { + assertThat(p("/hello/cat").resolveSibling(p("dog"))).isEqualTo(p("/hello/dog")); + assertThat(p("/").resolveSibling(p("dog"))).isEqualTo(p("dog")); + } + + @Test + public void testResolveSibling_preservesTrailingSlash() { + assertThat(p("/hello/cat").resolveSibling(p("dog/"))).isEqualTo(p("/hello/dog/")); + assertThat(p("/").resolveSibling(p("dog/"))).isEqualTo(p("dog/")); + } + + @Test + public void testRelativize() { + assertThat(p("/foo/bar/hop/dog").relativize(p("/foo/mop/top"))) + .isEqualTo(p("../../../mop/top")); + assertThat(p("/foo/bar/dog").relativize(p("/foo/mop/top"))).isEqualTo(p("../../mop/top")); + assertThat(p("/foo/bar/hop/dog").relativize(p("/foo/mop/top/../../mog"))) + .isEqualTo(p("../../../mop/top/../../mog")); + assertThat(p("/foo/bar/hop/dog").relativize(p("/foo/../mog"))).isEqualTo(p("../../../../mog")); + assertThat(p("").relativize(p("foo/mop/top/"))).isEqualTo(p("foo/mop/top/")); + } + + @Test + public void testRelativize_absoluteMismatch_notAllowed() { + thrown.expect(IllegalArgumentException.class); + p("/a/b/").relativize(p("")); + } + + @Test + public void testRelativize_preservesTrailingSlash() { + // This behavior actually diverges from sun.nio.fs.UnixPath: + // bsh % print(Paths.get("/a/b/").relativize(Paths.get("/etc/"))); + // ../../etc + assertThat(p("/foo/bar/hop/dog").relativize(p("/foo/../mog/"))) + .isEqualTo(p("../../../../mog/")); + assertThat(p("/a/b/").relativize(p("/etc/"))).isEqualTo(p("../../etc/")); + } + + @Test + public void testStartsWith() { + assertThat(p("/hi/there").startsWith(p("/hi/there"))).isTrue(); + assertThat(p("/hi/there").startsWith(p("/hi/therf"))).isFalse(); + assertThat(p("/hi/there").startsWith(p("/hi"))).isTrue(); + assertThat(p("/hi/there").startsWith(p("/hi/"))).isTrue(); + assertThat(p("/hi/there").startsWith(p("hi"))).isFalse(); + assertThat(p("/hi/there").startsWith(p("/"))).isTrue(); + assertThat(p("/hi/there").startsWith(p(""))).isFalse(); + assertThat(p("/a/b").startsWith(p("a/b/"))).isFalse(); + assertThat(p("/a/b/").startsWith(p("a/b/"))).isFalse(); + assertThat(p("/hi/there").startsWith(p(""))).isFalse(); + assertThat(p("").startsWith(p(""))).isTrue(); + } + + @Test + public void testStartsWith_comparesComponentsIndividually() { + assertThat(p("/hello").startsWith(p("/hell"))).isFalse(); + assertThat(p("/hello").startsWith(p("/hello"))).isTrue(); + } + + @Test + public void testEndsWith() { + assertThat(p("/hi/there").endsWith(p("there"))).isTrue(); + assertThat(p("/hi/there").endsWith(p("therf"))).isFalse(); + assertThat(p("/hi/there").endsWith(p("/blag/therf"))).isFalse(); + assertThat(p("/hi/there").endsWith(p("/hi/there"))).isTrue(); + assertThat(p("/hi/there").endsWith(p("/there"))).isFalse(); + assertThat(p("/human/that/you/cry").endsWith(p("that/you/cry"))).isTrue(); + assertThat(p("/human/that/you/cry").endsWith(p("that/you/cry/"))).isTrue(); + assertThat(p("/hi/there/").endsWith(p("/"))).isFalse(); + assertThat(p("/hi/there").endsWith(p(""))).isFalse(); + assertThat(p("").endsWith(p(""))).isTrue(); + } + + @Test + public void testEndsWith_comparesComponentsIndividually() { + assertThat(p("/hello").endsWith(p("lo"))).isFalse(); + assertThat(p("/hello").endsWith(p("hello"))).isTrue(); + } + + @Test + public void testGetParent() { + assertThat(p("").getParent()).isNull(); + assertThat(p("/").getParent()).isNull(); + assertThat(p("aaa/").getParent()).isNull(); + assertThat(p("aaa").getParent()).isNull(); + assertThat(p("/aaa/").getParent()).isEqualTo(p("/")); + assertThat(p("a/b/c").getParent()).isEqualTo(p("a/b/")); + assertThat(p("a/b/c/").getParent()).isEqualTo(p("a/b/")); + assertThat(p("a/b/").getParent()).isEqualTo(p("a/")); + } + + @Test + public void testGetRoot() { + assertThat(p("/hello").getRoot()).isEqualTo(p("/")); + assertThat(p("hello").getRoot()).isNull(); + } + + @Test + public void testGetFileName() { + assertThat(p("").getFileName()).isEqualTo(p("")); + assertThat(p("/").getFileName()).isNull(); + assertThat(p("/dark").getFileName()).isEqualTo(p("dark")); + assertThat(p("/angels/").getFileName()).isEqualTo(p("angels")); + } + + @Test + public void testEquals() { + assertThat(p("/a/").equals(p("/a/"))).isTrue(); + assertThat(p("/a/").equals(p("/b/"))).isFalse(); + assertThat(p("/b/").equals(p("/b"))).isFalse(); + assertThat(p("/b").equals(p("/b/"))).isFalse(); + assertThat(p("b").equals(p("/b"))).isFalse(); + assertThat(p("b").equals(p("b"))).isTrue(); + } + + @Test + public void testSplit() { + assertThat(p("").split().hasNext()).isFalse(); + assertThat(p("hi/there").split().hasNext()).isTrue(); + assertThat(p(p("hi/there").split().next())).isEqualTo(p("hi")); + } + + @Test + public void testToAbsolute() { + assertThat(p("lol").toAbsolutePath(UnixPath.ROOT_PATH)).isEqualTo(p("/lol")); + assertThat(p("lol/cat").toAbsolutePath(UnixPath.ROOT_PATH)).isEqualTo(p("/lol/cat")); + } + + @Test + public void testToAbsolute_withCurrentDirectory() { + assertThat(p("cat").toAbsolutePath(p("/lol"))).isEqualTo(p("/lol/cat")); + assertThat(p("cat").toAbsolutePath(p("/lol/"))).isEqualTo(p("/lol/cat")); + assertThat(p("/hi/there").toAbsolutePath(p("/lol"))).isEqualTo(p("/hi/there")); + } + + @Test + public void testToAbsolute_preservesTrailingSlash() { + assertThat(p("cat/").toAbsolutePath(p("/lol"))).isEqualTo(p("/lol/cat/")); + } + + @Test + public void testSubpath() { + assertThat(p("/eins/zwei/drei/vier").subpath(0, 1)).isEqualTo(p("eins")); + assertThat(p("/eins/zwei/drei/vier").subpath(0, 2)).isEqualTo(p("eins/zwei")); + assertThat(p("eins/zwei/drei/vier/").subpath(1, 4)).isEqualTo(p("zwei/drei/vier")); + assertThat(p("eins/zwei/drei/vier/").subpath(2, 4)).isEqualTo(p("drei/vier")); + } + + @Test + public void testSubpath_empty_returnsEmpty() { + assertThat(p("").subpath(0, 1)).isEqualTo(p("")); + } + + @Test + public void testSubpath_root_throwsIae() { + thrown.expect(IllegalArgumentException.class); + p("/").subpath(0, 1); + } + + @Test + public void testSubpath_negativeIndex_throwsIae() { + thrown.expect(IllegalArgumentException.class); + p("/eins/zwei/drei/vier").subpath(-1, 1); + } + + @Test + public void testSubpath_notEnoughElements_throwsIae() { + thrown.expect(IllegalArgumentException.class); + p("/eins/zwei/drei/vier").subpath(0, 5); + } + + @Test + public void testSubpath_beginAboveEnd_throwsIae() { + thrown.expect(IllegalArgumentException.class); + p("/eins/zwei/drei/vier").subpath(1, 0); + } + + @Test + public void testSubpath_beginAndEndEqual_throwsIae() { + thrown.expect(IllegalArgumentException.class); + p("/eins/zwei/drei/vier").subpath(0, 0); + } + + @Test + public void testNameCount() { + assertThat(p("").getNameCount()).isEqualTo(1); + assertThat(p("/").getNameCount()).isEqualTo(0); + assertThat(p("/hi/").getNameCount()).isEqualTo(1); + assertThat(p("/hi/yo").getNameCount()).isEqualTo(2); + assertThat(p("hi/yo").getNameCount()).isEqualTo(2); + } + + @Test + public void testNameCount_dontPermitEmptyComponents_emptiesGetIgnored() { + assertThat(p("hi//yo").getNameCount()).isEqualTo(2); + assertThat(p("//hi//yo//").getNameCount()).isEqualTo(2); + } + + @Test + public void testNameCount_permitEmptyComponents_emptiesGetCounted() { + assertThat(pp("hi//yo").getNameCount()).isEqualTo(3); + assertThat(pp("hi//yo/").getNameCount()).isEqualTo(4); + assertThat(pp("hi//yo//").getNameCount()).isEqualTo(5); + } + + @Test + public void testNameCount_permitEmptyComponents_rootComponentDoesntCount() { + assertThat(pp("hi/yo").getNameCount()).isEqualTo(2); + assertThat(pp("/hi/yo").getNameCount()).isEqualTo(2); + assertThat(pp("//hi/yo").getNameCount()).isEqualTo(3); + } + + @Test + public void testGetName() { + assertThat(p("").getName(0)).isEqualTo(p("")); + assertThat(p("/hi").getName(0)).isEqualTo(p("hi")); + assertThat(p("hi/there").getName(1)).isEqualTo(p("there")); + } + + @Test + public void testCompareTo() { + assertThat(p("/hi/there").compareTo(p("/hi/there"))).isEqualTo(0); + assertThat(p("/hi/there").compareTo(p("/hi/therf"))).isEqualTo(-1); + assertThat(p("/hi/there").compareTo(p("/hi/therd"))).isEqualTo(1); + } + + @Test + public void testCompareTo_dontPermitEmptyComponents_emptiesGetIgnored() { + assertThat(p("a/b").compareTo(p("a//b"))).isEqualTo(0); + } + + @Test + public void testCompareTo_permitEmptyComponents_behaviorChanges() { + assertThat(p("a/b").compareTo(pp("a//b"))).isEqualTo(1); + assertThat(pp("a/b").compareTo(pp("a//b"))).isEqualTo(1); + } + + @Test + public void testCompareTo_comparesComponentsIndividually() { + assumeTrue('.' < '/'); + assertThat("hi./there".compareTo("hi/there")).isEqualTo(-1); + assertThat("hi.".compareTo("hi")).isEqualTo(1); + assertThat(p("hi./there").compareTo(p("hi/there"))).isEqualTo(1); + assertThat(p("hi./there").compareTo(p("hi/there"))).isEqualTo(1); + assumeTrue('0' > '/'); + assertThat("hi0/there".compareTo("hi/there")).isEqualTo(1); + assertThat("hi0".compareTo("hi")).isEqualTo(1); + assertThat(p("hi0/there").compareTo(p("hi/there"))).isEqualTo(1); + } + + @Test + public void testSeemsLikeADirectory() { + assertThat(p("a").seemsLikeADirectory()).isFalse(); + assertThat(p("a.").seemsLikeADirectory()).isFalse(); + assertThat(p("a..").seemsLikeADirectory()).isFalse(); + assertThat(p("").seemsLikeADirectory()).isTrue(); + assertThat(p("/").seemsLikeADirectory()).isTrue(); + assertThat(p(".").seemsLikeADirectory()).isTrue(); + assertThat(p("/.").seemsLikeADirectory()).isTrue(); + assertThat(p("..").seemsLikeADirectory()).isTrue(); + assertThat(p("/..").seemsLikeADirectory()).isTrue(); + } + + @Test + public void testEquals_equalsTester() { + new EqualsTester() + .addEqualityGroup(p("/lol"), p("/lol")) + .addEqualityGroup(p("/lol//"), p("/lol//")) + .addEqualityGroup(p("dust")) + .testEquals(); + } + + @Test + public void testNullness() throws Exception { + NullPointerTester tester = new NullPointerTester(); + tester.ignore(UnixPath.class.getMethod("equals", Object.class)); + tester.testAllPublicStaticMethods(UnixPath.class); + tester.testAllPublicInstanceMethods(p("solo")); + } + + private static UnixPath p(String path) { + return UnixPath.getPath(false, path); + } + + private static UnixPath pp(String path) { + return UnixPath.getPath(true, path); + } +} diff --git a/gcloud-java-contrib/pom.xml b/gcloud-java-contrib/pom.xml index 5d5739781727..b191e08d6297 100644 --- a/gcloud-java-contrib/pom.xml +++ b/gcloud-java-contrib/pom.xml @@ -1,9 +1,8 @@ 4.0.0 - com.google.gcloud gcloud-java-contrib - jar + pom GCloud Java contributions Contains packages that provide higher-level abstraction/functionality for common gcloud-java use cases. @@ -11,11 +10,14 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT gcloud-java-contrib + + gcloud-java-nio + ${project.groupId} diff --git a/gcloud-java-core/README.md b/gcloud-java-core/README.md index 9063bebebbef..fc5f481f8ec3 100644 --- a/gcloud-java-core/README.md +++ b/gcloud-java-core/README.md @@ -6,6 +6,8 @@ This module provides common functionality required by service-specific modules o [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-core.svg)](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-core.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/package-summary.html) @@ -17,16 +19,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-core - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-core:0.1.3' +compile 'com.google.gcloud:gcloud-java-core:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-core" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java-core" % "0.1.5" ``` Troubleshooting diff --git a/gcloud-java-core/pom.xml b/gcloud-java-core/pom.xml index 7373b40abc75..6d0ed675b423 100644 --- a/gcloud-java-core/pom.xml +++ b/gcloud-java-core/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-core jar GCloud Java core @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT gcloud-java-core @@ -36,24 +35,24 @@ com.google.http-client google-http-client - 1.20.0 + 1.21.0 compile com.google.oauth-client google-oauth-client - 1.20.0 + 1.21.0 compile com.google.guava guava - 18.0 + 19.0 com.google.api-client google-api-client-appengine - 1.20.0 + 1.21.0 compile @@ -65,7 +64,7 @@ com.google.http-client google-http-client-jackson - 1.20.0 + 1.21.0 compile @@ -83,19 +82,19 @@ joda-time joda-time - 2.8.2 + 2.9.2 compile org.json json - 20090211 + 20151123 compile org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/AuthCredentials.java b/gcloud-java-core/src/main/java/com/google/gcloud/AuthCredentials.java index fc5d74d0896c..6f9e09ca04bc 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/AuthCredentials.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/AuthCredentials.java @@ -132,6 +132,12 @@ public RestorableState capture() { } } + /** + * Represents service account credentials. + * + * @see + * User accounts and service accounts + */ public static class ServiceAccountAuthCredentials extends AuthCredentials { private final String account; @@ -195,6 +201,14 @@ public RestorableState capture() { } } + /** + * Represents Application Default Credentials, which are credentials that are inferred from the + * runtime environment. + * + * @see + * Google Application Default Credentials + */ public static class ApplicationDefaultAuthCredentials extends AuthCredentials { private GoogleCredentials googleCredentials; @@ -243,6 +257,38 @@ public RestorableState capture() { } } + /** + * A placeholder for credentials to signify that requests sent to the server should not be + * authenticated. This is typically useful when using the local service emulators, such as + * {@code LocalGcdHelper} and {@code LocalResourceManagerHelper}. + */ + public static class NoAuthCredentials extends AuthCredentials { + + private static final AuthCredentials INSTANCE = new NoAuthCredentials(); + private static final NoAuthCredentialsState STATE = new NoAuthCredentialsState(); + + private static class NoAuthCredentialsState + implements RestorableState, Serializable { + + private static final long serialVersionUID = -4022100563954640465L; + + @Override + public AuthCredentials restore() { + return INSTANCE; + } + } + + @Override + public GoogleCredentials credentials() { + return null; + } + + @Override + public RestorableState capture() { + return STATE; + } + } + public abstract GoogleCredentials credentials(); public static AuthCredentials createForAppEngine() { @@ -281,6 +327,15 @@ public static ServiceAccountAuthCredentials createFor(String account, PrivateKey return new ServiceAccountAuthCredentials(account, privateKey); } + /** + * Creates a placeholder denoting that no credentials should be used. This is typically useful + * when using the local service emulators, such as {@code LocalGcdHelper} and + * {@code LocalResourceManagerHelper}. + */ + public static AuthCredentials noAuth() { + return NoAuthCredentials.INSTANCE; + } + /** * Creates Service Account Credentials given a stream for credentials in JSON format. * diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/BaseServiceException.java b/gcloud-java-core/src/main/java/com/google/gcloud/BaseServiceException.java index 579340f1256e..365243904436 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/BaseServiceException.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/BaseServiceException.java @@ -97,13 +97,17 @@ public BaseServiceException(IOException exception, boolean idempotent) { String debugInfo = null; if (exception instanceof GoogleJsonResponseException) { GoogleJsonError jsonError = ((GoogleJsonResponseException) exception).getDetails(); - Error error = error(jsonError); - code = error.code; - reason = error.reason; - if (reason != null) { - GoogleJsonError.ErrorInfo errorInfo = jsonError.getErrors().get(0); - location = errorInfo.getLocation(); - debugInfo = (String) errorInfo.get("debugInfo"); + if (jsonError != null) { + Error error = error(jsonError); + code = error.code; + reason = error.reason; + if (reason != null) { + GoogleJsonError.ErrorInfo errorInfo = jsonError.getErrors().get(0); + location = errorInfo.getLocation(); + debugInfo = (String) errorInfo.get("debugInfo"); + } + } else { + code = ((GoogleJsonResponseException) exception).getStatusCode(); } } this.code = code; @@ -207,7 +211,10 @@ protected static Error error(GoogleJsonError error) { protected static String message(IOException exception) { if (exception instanceof GoogleJsonResponseException) { - return ((GoogleJsonResponseException) exception).getDetails().getMessage(); + GoogleJsonError details = ((GoogleJsonResponseException) exception).getDetails(); + if (details != null) { + return details.getMessage(); + } } return exception.getMessage(); } diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java b/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java index e05383a65826..1d18a5a27e81 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java @@ -21,6 +21,7 @@ import java.io.IOException; import java.io.Serializable; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Arrays; import java.util.Objects; @@ -114,9 +115,9 @@ private void flush() { } } - private void validateOpen() throws IOException { + private void validateOpen() throws ClosedChannelException { if (!isOpen) { - throw new IOException("stream is closed"); + throw new ClosedChannelException(); } } diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java b/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java new file mode 100644 index 000000000000..748eaba2ab4c --- /dev/null +++ b/gcloud-java-core/src/main/java/com/google/gcloud/IamPolicy.java @@ -0,0 +1,256 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud; + +import static com.google.common.base.Preconditions.checkArgument; + +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; + +import java.io.Serializable; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +/** + * Base class for Identity and Access Management (IAM) policies. IAM policies are used to specify + * access settings for Cloud Platform resources. A policy is a map of bindings. A binding assigns + * a set of identities to a role, where the identities can be user accounts, Google groups, Google + * domains, and service accounts. A role is a named list of permissions defined by IAM. + * + * @param the data type of roles (should be serializable) + * @see Policy + */ +public abstract class IamPolicy implements Serializable { + + private static final long serialVersionUID = 1114489978726897720L; + + private final Map> bindings; + private final String etag; + private final Integer version; + + /** + * Builder for an IAM Policy. + * + * @param the data type of roles + * @param the subclass extending this abstract builder + */ + public abstract static class Builder> { + + private final Map> bindings = new HashMap<>(); + private String etag; + private Integer version; + + /** + * Constructor for IAM Policy builder. + */ + protected Builder() {} + + /** + * Replaces the builder's map of bindings with the given map of bindings. + * + * @throws IllegalArgumentException if the provided map is null or contain any null values + */ + public final B bindings(Map> bindings) { + checkArgument(bindings != null, "The provided map of bindings cannot be null."); + for (Map.Entry> binding : bindings.entrySet()) { + verifyBinding(binding.getKey(), binding.getValue()); + } + this.bindings.clear(); + for (Map.Entry> binding : bindings.entrySet()) { + this.bindings.put(binding.getKey(), new HashSet(binding.getValue())); + } + return self(); + } + + /** + * Adds a binding to the policy. + * + * @throws IllegalArgumentException if the policy already contains a binding with the same role + * or if the role or any identities are null + */ + public final B addBinding(R role, Set identities) { + verifyBinding(role, identities); + checkArgument(!bindings.containsKey(role), + "The policy already contains a binding with the role " + role.toString() + "."); + bindings.put(role, new HashSet(identities)); + return self(); + } + + /** + * Adds a binding to the policy. + * + * @throws IllegalArgumentException if the policy already contains a binding with the same role + * or if the role or any identities are null + */ + public final B addBinding(R role, Identity first, Identity... others) { + HashSet identities = new HashSet<>(); + identities.add(first); + identities.addAll(Arrays.asList(others)); + return addBinding(role, identities); + } + + private void verifyBinding(R role, Collection identities) { + checkArgument(role != null, "The role cannot be null."); + verifyIdentities(identities); + } + + private void verifyIdentities(Collection identities) { + checkArgument(identities != null, "A role cannot be assigned to a null set of identities."); + checkArgument(!identities.contains(null), "Null identities are not permitted."); + } + + /** + * Removes the binding associated with the specified role. + */ + public final B removeBinding(R role) { + bindings.remove(role); + return self(); + } + + /** + * Adds one or more identities to an existing binding. + * + * @throws IllegalArgumentException if the policy doesn't contain a binding with the specified + * role or any identities are null + */ + public final B addIdentity(R role, Identity first, Identity... others) { + checkArgument(bindings.containsKey(role), + "The policy doesn't contain the role " + role.toString() + "."); + List toAdd = new LinkedList<>(); + toAdd.add(first); + toAdd.addAll(Arrays.asList(others)); + verifyIdentities(toAdd); + bindings.get(role).addAll(toAdd); + return self(); + } + + /** + * Removes one or more identities from an existing binding. + * + * @throws IllegalArgumentException if the policy doesn't contain a binding with the specified + * role + */ + public final B removeIdentity(R role, Identity first, Identity... others) { + checkArgument(bindings.containsKey(role), + "The policy doesn't contain the role " + role.toString() + "."); + bindings.get(role).remove(first); + bindings.get(role).removeAll(Arrays.asList(others)); + return self(); + } + + /** + * Sets the policy's etag. + * + *

Etags are used for optimistic concurrency control as a way to help prevent simultaneous + * updates of a policy from overwriting each other. It is strongly suggested that systems make + * use of the etag in the read-modify-write cycle to perform policy updates in order to avoid + * race conditions. An etag is returned in the response to getIamPolicy, and systems are + * expected to put that etag in the request to setIamPolicy to ensure that their change will be + * applied to the same version of the policy. If no etag is provided in the call to + * setIamPolicy, then the existing policy is overwritten blindly. + */ + protected final B etag(String etag) { + this.etag = etag; + return self(); + } + + /** + * Sets the version of the policy. The default version is 0, meaning only the "owner", "editor", + * and "viewer" roles are permitted. If the version is 1, you may also use other roles. + */ + protected final B version(Integer version) { + this.version = version; + return self(); + } + + @SuppressWarnings("unchecked") + private B self() { + return (B) this; + } + + public abstract IamPolicy build(); + } + + protected IamPolicy(Builder> builder) { + ImmutableMap.Builder> bindingsBuilder = ImmutableMap.builder(); + for (Map.Entry> binding : builder.bindings.entrySet()) { + bindingsBuilder.put(binding.getKey(), ImmutableSet.copyOf(binding.getValue())); + } + this.bindings = bindingsBuilder.build(); + this.etag = builder.etag; + this.version = builder.version; + } + + /** + * Returns a builder containing the properties of this IAM Policy. + */ + public abstract Builder> toBuilder(); + + /** + * The map of bindings that comprises the policy. + */ + public Map> bindings() { + return bindings; + } + + /** + * The policy's etag. + * + *

Etags are used for optimistic concurrency control as a way to help prevent simultaneous + * updates of a policy from overwriting each other. It is strongly suggested that systems make + * use of the etag in the read-modify-write cycle to perform policy updates in order to avoid + * race conditions. An etag is returned in the response to getIamPolicy, and systems are + * expected to put that etag in the request to setIamPolicy to ensure that their change will be + * applied to the same version of the policy. If no etag is provided in the call to + * setIamPolicy, then the existing policy is overwritten blindly. + */ + public String etag() { + return etag; + } + + /** + * Sets the version of the policy. The default version is 0, meaning only the "owner", "editor", + * and "viewer" roles are permitted. If the version is 1, you may also use other roles. + */ + public Integer version() { + return version; + } + + @Override + public final int hashCode() { + return Objects.hash(getClass(), bindings, etag, version); + } + + @Override + public final boolean equals(Object obj) { + if (obj == null || !getClass().equals(obj.getClass())) { + return false; + } + @SuppressWarnings("rawtypes") + IamPolicy other = (IamPolicy) obj; + return Objects.equals(bindings, other.bindings()) + && Objects.equals(etag, other.etag()) + && Objects.equals(version, other.version()); + } +} diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java b/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java new file mode 100644 index 000000000000..d1644198f759 --- /dev/null +++ b/gcloud-java-core/src/main/java/com/google/gcloud/Identity.java @@ -0,0 +1,225 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud; + +import static com.google.common.base.Preconditions.checkNotNull; + +import com.google.common.base.CaseFormat; + +import java.io.Serializable; +import java.util.Objects; + +/** + * An identity in an {@link IamPolicy}. The following types of identities are permitted in IAM + * policies: + *

    + *
  • Google account + *
  • Service account + *
  • Google group + *
  • Google Apps domain + *
+ * + *

There are also two special identities that represent all users and all Google-authenticated + * accounts. + * + * @see Concepts + * related to identity + */ +public final class Identity implements Serializable { + + private static final long serialVersionUID = -8181841964597657446L; + + private final Type type; + private final String id; + + /** + * The types of IAM identities. + */ + public enum Type { + + /** + * Represents anyone who is on the internet; with or without a Google account. + */ + ALL_USERS, + + /** + * Represents anyone who is authenticated with a Google account or a service account. + */ + ALL_AUTHENTICATED_USERS, + + /** + * Represents a specific Google account. + */ + USER, + + /** + * Represents a service account. + */ + SERVICE_ACCOUNT, + + /** + * Represents a Google group. + */ + GROUP, + + /** + * Represents all the users of a Google Apps domain name. + */ + DOMAIN + } + + private Identity(Type type, String id) { + this.type = type; + this.id = id; + } + + public Type type() { + return type; + } + + /** + * Returns the string identifier for this identity. The id corresponds to: + *

    + *
  • email address (for identities of type {@code USER}, {@code SERVICE_ACCOUNT}, and + * {@code GROUP}) + *
  • domain (for identities of type {@code DOMAIN}) + *
  • {@code null} (for identities of type {@code ALL_USERS} and + * {@code ALL_AUTHENTICATED_USERS}) + *
+ */ + public String id() { + return id; + } + + /** + * Returns a new identity representing anyone who is on the internet; with or without a Google + * account. + */ + public static Identity allUsers() { + return new Identity(Type.ALL_USERS, null); + } + + /** + * Returns a new identity representing anyone who is authenticated with a Google account or a + * service account. + */ + public static Identity allAuthenticatedUsers() { + return new Identity(Type.ALL_AUTHENTICATED_USERS, null); + } + + /** + * Returns a new user identity. + * + * @param email An email address that represents a specific Google account. For example, + * alice@gmail.com or joe@example.com. + */ + public static Identity user(String email) { + return new Identity(Type.USER, checkNotNull(email)); + } + + /** + * Returns a new service account identity. + * + * @param email An email address that represents a service account. For example, + * my-other-app@appspot.gserviceaccount.com. + */ + public static Identity serviceAccount(String email) { + return new Identity(Type.SERVICE_ACCOUNT, checkNotNull(email)); + } + + /** + * Returns a new group identity. + * + * @param email An email address that represents a Google group. For example, + * admins@example.com. + */ + public static Identity group(String email) { + return new Identity(Type.GROUP, checkNotNull(email)); + } + + /** + * Returns a new domain identity. + * + * @param domain A Google Apps domain name that represents all the users of that domain. For + * example, google.com or example.com. + */ + public static Identity domain(String domain) { + return new Identity(Type.DOMAIN, checkNotNull(domain)); + } + + @Override + public int hashCode() { + return Objects.hash(id, type); + } + + @Override + public boolean equals(Object obj) { + if (!(obj instanceof Identity)) { + return false; + } + Identity other = (Identity) obj; + return Objects.equals(id, other.id()) && Objects.equals(type, other.type()); + } + + /** + * Returns the string value associated with the identity. Used primarily for converting from + * {@code Identity} objects to strings for protobuf-generated policies. + */ + public String strValue() { + switch (type) { + case ALL_USERS: + return "allUsers"; + case ALL_AUTHENTICATED_USERS: + return "allAuthenticatedUsers"; + case USER: + return "user:" + id; + case SERVICE_ACCOUNT: + return "serviceAccount:" + id; + case GROUP: + return "group:" + id; + case DOMAIN: + return "domain:" + id; + default: + throw new IllegalStateException("Unexpected identity type: " + type); + } + } + + /** + * Converts a string to an {@code Identity}. Used primarily for converting protobuf-generated + * policy identities to {@code Identity} objects. + */ + public static Identity valueOf(String identityStr) { + String[] info = identityStr.split(":"); + Type type = Type.valueOf(CaseFormat.LOWER_CAMEL.to(CaseFormat.UPPER_UNDERSCORE, info[0])); + switch (type) { + case ALL_USERS: + return Identity.allUsers(); + case ALL_AUTHENTICATED_USERS: + return Identity.allAuthenticatedUsers(); + case USER: + return Identity.user(info[1]); + case SERVICE_ACCOUNT: + return Identity.serviceAccount(info[1]); + case GROUP: + return Identity.group(info[1]); + case DOMAIN: + return Identity.domain(info[1]); + default: + throw new IllegalStateException("Unexpected identity type " + type); + } + } +} diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java b/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java index 90633c70046f..0b573522e370 100644 --- a/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java +++ b/gcloud-java-core/src/main/java/com/google/gcloud/Restorable.java @@ -21,14 +21,14 @@ * *

* A typical capture usage: - *

  {@code
+ * 
 {@code
  * X restorableObj; // X instanceof Restorable
  * RestorableState state = restorableObj.capture();
  * .. persist state
  * }
* * A typical restore usage: - *
  {@code
+ * 
 {@code
  * RestorableState state = ... // read from persistence
  * X restorableObj = state.restore();
  * ...
diff --git a/gcloud-java-core/src/main/java/com/google/gcloud/ServiceOptions.java b/gcloud-java-core/src/main/java/com/google/gcloud/ServiceOptions.java
index 31e543809464..d45069434a26 100644
--- a/gcloud-java-core/src/main/java/com/google/gcloud/ServiceOptions.java
+++ b/gcloud-java-core/src/main/java/com/google/gcloud/ServiceOptions.java
@@ -523,9 +523,10 @@ public RetryParams retryParams() {
    * options.
    */
   public HttpRequestInitializer httpRequestInitializer() {
-    final HttpRequestInitializer delegate = authCredentials() != null
-        ? new HttpCredentialsAdapter(authCredentials().credentials().createScoped(scopes()))
-        : null;
+    final HttpRequestInitializer delegate =
+        authCredentials() != null && authCredentials.credentials() != null
+            ? new HttpCredentialsAdapter(authCredentials().credentials().createScoped(scopes()))
+            : null;
     return new HttpRequestInitializer() {
       @Override
       public void initialize(HttpRequest httpRequest) throws IOException {
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
index e49a17b019e0..6d5306a3bc7f 100644
--- a/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/BaseWriteChannelTest.java
@@ -32,6 +32,7 @@
 import java.io.IOException;
 import java.io.Serializable;
 import java.nio.ByteBuffer;
+import java.nio.channels.ClosedChannelException;
 import java.util.Arrays;
 import java.util.Random;
 
@@ -102,8 +103,7 @@ public void testClose() throws IOException {
   @Test
   public void testValidateOpen() throws IOException {
     channel.close();
-    thrown.expect(IOException.class);
-    thrown.expectMessage("stream is closed");
+    thrown.expect(ClosedChannelException.class);
     channel.write(ByteBuffer.allocate(42));
   }
 
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java
new file mode 100644
index 000000000000..db0935c4766d
--- /dev/null
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/IamPolicyTest.java
@@ -0,0 +1,180 @@
+/*
+ * Copyright 2016 Google Inc. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.gcloud;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.Set;
+
+public class IamPolicyTest {
+
+  private static final Identity ALL_USERS = Identity.allUsers();
+  private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers();
+  private static final Identity USER = Identity.user("abc@gmail.com");
+  private static final Identity SERVICE_ACCOUNT =
+      Identity.serviceAccount("service-account@gmail.com");
+  private static final Identity GROUP = Identity.group("group@gmail.com");
+  private static final Identity DOMAIN = Identity.domain("google.com");
+  private static final Map> BINDINGS = ImmutableMap.of(
+      "viewer",
+      ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS),
+      "editor",
+      ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN));
+  private static final PolicyImpl SIMPLE_POLICY = PolicyImpl.builder()
+      .addBinding("viewer", ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS))
+      .addBinding("editor", ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN))
+      .build();
+  private static final PolicyImpl FULL_POLICY =
+      new PolicyImpl.Builder(SIMPLE_POLICY.bindings(), "etag", 1).build();
+
+  static class PolicyImpl extends IamPolicy {
+
+    static class Builder extends IamPolicy.Builder {
+
+      private Builder() {}
+
+      private Builder(Map> bindings, String etag, Integer version) {
+        bindings(bindings).etag(etag).version(version);
+      }
+
+      @Override
+      public PolicyImpl build() {
+        return new PolicyImpl(this);
+      }
+    }
+
+    PolicyImpl(Builder builder) {
+      super(builder);
+    }
+
+    @Override
+    public Builder toBuilder() {
+      return new Builder(bindings(), etag(), version());
+    }
+
+    static Builder builder() {
+      return new Builder();
+    }
+  }
+
+  @Test
+  public void testBuilder() {
+    assertEquals(BINDINGS, FULL_POLICY.bindings());
+    assertEquals("etag", FULL_POLICY.etag());
+    assertEquals(1, FULL_POLICY.version().intValue());
+    Map> editorBinding =
+        ImmutableMap.>builder().put("editor", BINDINGS.get("editor")).build();
+    PolicyImpl policy = FULL_POLICY.toBuilder().bindings(editorBinding).build();
+    assertEquals(editorBinding, policy.bindings());
+    assertEquals("etag", policy.etag());
+    assertEquals(1, policy.version().intValue());
+    policy = SIMPLE_POLICY.toBuilder().removeBinding("editor").build();
+    assertEquals(ImmutableMap.of("viewer", BINDINGS.get("viewer")), policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    policy = policy.toBuilder()
+        .removeIdentity("viewer", USER, ALL_USERS)
+        .addIdentity("viewer", DOMAIN, GROUP)
+        .build();
+    assertEquals(ImmutableMap.of("viewer", ImmutableSet.of(SERVICE_ACCOUNT, DOMAIN, GROUP)),
+        policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    policy = PolicyImpl.builder().addBinding("owner", USER, SERVICE_ACCOUNT).build();
+    assertEquals(
+        ImmutableMap.of("owner", ImmutableSet.of(USER, SERVICE_ACCOUNT)), policy.bindings());
+    assertNull(policy.etag());
+    assertNull(policy.version());
+    try {
+      SIMPLE_POLICY.toBuilder().addBinding("viewer", USER);
+      fail("Should have failed due to duplicate role.");
+    } catch (IllegalArgumentException e) {
+      assertEquals("The policy already contains a binding with the role viewer.", e.getMessage());
+    }
+    try {
+      SIMPLE_POLICY.toBuilder().addBinding("editor", ImmutableSet.of(USER));
+      fail("Should have failed due to duplicate role.");
+    } catch (IllegalArgumentException e) {
+      assertEquals("The policy already contains a binding with the role editor.", e.getMessage());
+    }
+  }
+
+  @Test
+  public void testEqualsHashCode() {
+    assertNotNull(FULL_POLICY);
+    PolicyImpl emptyPolicy = PolicyImpl.builder().build();
+    AnotherPolicyImpl anotherPolicy = new AnotherPolicyImpl.Builder().build();
+    assertNotEquals(emptyPolicy, anotherPolicy);
+    assertNotEquals(emptyPolicy.hashCode(), anotherPolicy.hashCode());
+    assertNotEquals(FULL_POLICY, SIMPLE_POLICY);
+    assertNotEquals(FULL_POLICY.hashCode(), SIMPLE_POLICY.hashCode());
+    PolicyImpl copy = SIMPLE_POLICY.toBuilder().build();
+    assertEquals(SIMPLE_POLICY, copy);
+    assertEquals(SIMPLE_POLICY.hashCode(), copy.hashCode());
+  }
+
+  @Test
+  public void testBindings() {
+    assertTrue(PolicyImpl.builder().build().bindings().isEmpty());
+    assertEquals(BINDINGS, SIMPLE_POLICY.bindings());
+  }
+
+  @Test
+  public void testEtag() {
+    assertNull(SIMPLE_POLICY.etag());
+    assertEquals("etag", FULL_POLICY.etag());
+  }
+
+  @Test
+  public void testVersion() {
+    assertNull(SIMPLE_POLICY.version());
+    assertEquals(1, FULL_POLICY.version().intValue());
+  }
+
+  static class AnotherPolicyImpl extends IamPolicy {
+
+    static class Builder extends IamPolicy.Builder {
+
+      private Builder() {}
+
+      @Override
+      public AnotherPolicyImpl build() {
+        return new AnotherPolicyImpl(this);
+      }
+    }
+
+    AnotherPolicyImpl(Builder builder) {
+      super(builder);
+    }
+
+    @Override
+    public Builder toBuilder() {
+      return new Builder();
+    }
+  }
+}
diff --git a/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java b/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java
new file mode 100644
index 000000000000..828f1c839431
--- /dev/null
+++ b/gcloud-java-core/src/test/java/com/google/gcloud/IdentityTest.java
@@ -0,0 +1,105 @@
+/*
+ * Copyright 2016 Google Inc. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.google.gcloud;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+import org.junit.Test;
+
+public class IdentityTest {
+
+  private static final Identity ALL_USERS = Identity.allUsers();
+  private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers();
+  private static final Identity USER = Identity.user("abc@gmail.com");
+  private static final Identity SERVICE_ACCOUNT =
+      Identity.serviceAccount("service-account@gmail.com");
+  private static final Identity GROUP = Identity.group("group@gmail.com");
+  private static final Identity DOMAIN = Identity.domain("google.com");
+
+  @Test
+  public void testAllUsers() {
+    assertEquals(Identity.Type.ALL_USERS, ALL_USERS.type());
+    assertNull(ALL_USERS.id());
+  }
+
+  @Test
+  public void testAllAuthenticatedUsers() {
+    assertEquals(Identity.Type.ALL_AUTHENTICATED_USERS, ALL_AUTH_USERS.type());
+    assertNull(ALL_AUTH_USERS.id());
+  }
+
+  @Test
+  public void testUser() {
+    assertEquals(Identity.Type.USER, USER.type());
+    assertEquals("abc@gmail.com", USER.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testUserNullEmail() {
+    Identity.user(null);
+  }
+
+  @Test
+  public void testServiceAccount() {
+    assertEquals(Identity.Type.SERVICE_ACCOUNT, SERVICE_ACCOUNT.type());
+    assertEquals("service-account@gmail.com", SERVICE_ACCOUNT.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testServiceAccountNullEmail() {
+    Identity.serviceAccount(null);
+  }
+
+  @Test
+  public void testGroup() {
+    assertEquals(Identity.Type.GROUP, GROUP.type());
+    assertEquals("group@gmail.com", GROUP.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testGroupNullEmail() {
+    Identity.group(null);
+  }
+
+  @Test
+  public void testDomain() {
+    assertEquals(Identity.Type.DOMAIN, DOMAIN.type());
+    assertEquals("google.com", DOMAIN.id());
+  }
+
+  @Test(expected = NullPointerException.class)
+  public void testDomainNullId() {
+    Identity.domain(null);
+  }
+
+  @Test
+  public void testIdentityToAndFromPb() {
+    compareIdentities(ALL_USERS, Identity.valueOf(ALL_USERS.strValue()));
+    compareIdentities(ALL_AUTH_USERS, Identity.valueOf(ALL_AUTH_USERS.strValue()));
+    compareIdentities(USER, Identity.valueOf(USER.strValue()));
+    compareIdentities(SERVICE_ACCOUNT, Identity.valueOf(SERVICE_ACCOUNT.strValue()));
+    compareIdentities(GROUP, Identity.valueOf(GROUP.strValue()));
+    compareIdentities(DOMAIN, Identity.valueOf(DOMAIN.strValue()));
+  }
+
+  private void compareIdentities(Identity expected, Identity actual) {
+    assertEquals(expected, actual);
+    assertEquals(expected.type(), actual.type());
+    assertEquals(expected.id(), actual.id());
+  }
+}
diff --git a/gcloud-java-datastore/README.md b/gcloud-java-datastore/README.md
index 7eae00f2ad3f..0d89a0a07e3e 100644
--- a/gcloud-java-datastore/README.md
+++ b/gcloud-java-datastore/README.md
@@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Datastore] (https://cloud.google.com/dat
 [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java)
 [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master)
 [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-datastore.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-datastore.svg)
+[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java)
+[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969)
 
 -  [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
 -  [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/datastore/package-summary.html)
@@ -20,21 +22,21 @@ If you are using Maven, add this to your pom.xml file
 
   com.google.gcloud
   gcloud-java-datastore
-  0.1.3
+  0.1.5
 
 ```
 If you are using Gradle, add this to your dependencies
 ```Groovy
-compile 'com.google.gcloud:gcloud-java-datastore:0.1.3'
+compile 'com.google.gcloud:gcloud-java-datastore:0.1.5'
 ```
 If you are using SBT, add this to your dependencies
 ```Scala
-libraryDependencies += "com.google.gcloud" % "gcloud-java-datastore" % "0.1.3"
+libraryDependencies += "com.google.gcloud" % "gcloud-java-datastore" % "0.1.5"
 ```
 
 Example Application
 --------------------
-[`DatastoreExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java) is a simple command line interface for the Cloud Datastore.  Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/DatastoreExample.html).
+[`DatastoreExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java) is a simple command line interface for the Cloud Datastore.  Read more about using the application on the [`DatastoreExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/datastore/DatastoreExample.html).
 
 Authentication
 --------------
@@ -134,67 +136,12 @@ Cloud Datastore relies on indexing to run queries. Indexing is turned on by defa
 
 #### Complete source code
 
-Here we put together all the code shown above into one program.  This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, move this code to your application's servlet class and print the query output to the webpage instead of `System.out`.
-
-```java
-import com.google.gcloud.datastore.Datastore;
-import com.google.gcloud.datastore.DatastoreOptions;
-import com.google.gcloud.datastore.Entity;
-import com.google.gcloud.datastore.Key;
-import com.google.gcloud.datastore.KeyFactory;
-import com.google.gcloud.datastore.Query;
-import com.google.gcloud.datastore.QueryResults;
-import com.google.gcloud.datastore.StructuredQuery;
-import com.google.gcloud.datastore.StructuredQuery.PropertyFilter;
-
-public class GcloudDatastoreExample {
-
-  public static void main(String[] args) {
-    // Create datastore service object.
-    // By default, credentials are inferred from the runtime environment.
-    Datastore datastore = DatastoreOptions.defaultInstance().service();
-
-    // Add an entity to Datastore
-    KeyFactory keyFactory = datastore.newKeyFactory().kind("Person");
-    Key key = keyFactory.newKey("john.doe@gmail.com");
-    Entity entity = Entity.builder(key)
-        .set("name", "John Doe")
-        .set("age", 51)
-        .set("favorite_food", "pizza")
-        .build();
-    datastore.put(entity);
-
-    // Get an entity from Datastore
-    Entity johnEntity = datastore.get(key);
-
-    // Add a couple more entities to make the query results more interesting
-    Key janeKey = keyFactory.newKey("jane.doe@gmail.com");
-    Entity janeEntity = Entity.builder(janeKey)
-        .set("name", "Jane Doe")
-        .set("age", 44)
-        .set("favorite_food", "pizza")
-        .build();
-    Key joeKey = keyFactory.newKey("joe.shmoe@gmail.com");
-    Entity joeEntity = Entity.builder(joeKey)
-        .set("name", "Joe Shmoe")
-        .set("age", 27)
-        .set("favorite_food", "sushi")
-        .build();
-    datastore.put(janeEntity, joeEntity);
-
-    // Run a query
-    Query query = Query.entityQueryBuilder()
-        .kind("Person")
-        .filter(PropertyFilter.eq("favorite_food", "pizza"))
-        .build();
-    QueryResults results = datastore.run(query);
-    while (results.hasNext()) {
-      Entity currentEntity = results.next();
-      System.out.println(currentEntity.getString("name") + ", you're invited to a pizza party!");
-    }
-  }
-}
-```
+In
+[AddEntitiesAndRunQuery.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java)
+we put together all the code shown above into one program. The program assumes that you are
+running on Compute Engine or from your own desktop. To run the example on App Engine, simply move
+the code from the main method to your application's servlet class and change the print statements to
+display on your webpage.
 
 Troubleshooting
 ---------------
diff --git a/gcloud-java-datastore/pom.xml b/gcloud-java-datastore/pom.xml
index 2a5d8e52e640..977b6db22b14 100644
--- a/gcloud-java-datastore/pom.xml
+++ b/gcloud-java-datastore/pom.xml
@@ -1,7 +1,6 @@
 
 
   4.0.0
-  com.google.gcloud
   gcloud-java-datastore
   jar
   GCloud Java datastore
@@ -11,7 +10,7 @@
   
     com.google.gcloud
     gcloud-java-pom
-    0.1.4-SNAPSHOT
+    0.1.6-SNAPSHOT
   
   
     gcloud-java-datastore
@@ -25,7 +24,7 @@
     
       com.google.apis
       google-api-services-datastore-protobuf
-      v1beta2-rev1-2.1.2
+      v1beta2-rev1-4.0.0
       compile
       
         
@@ -43,7 +42,7 @@
     
       org.easymock
       easymock
-      3.3
+      3.4
       test
     
   
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
index d674a5e242ad..20c0b13e5001 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseEntity.java
@@ -33,6 +33,7 @@
 import com.google.protobuf.InvalidProtocolBufferException;
 
 import java.util.HashMap;
+import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -128,61 +129,286 @@ public B remove(String name) {
       return self();
     }
 
+    /**
+     * Sets a property.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Value value) {
       properties.put(name, value);
       return self();
     }
 
+    /**
+     * Sets a property of type {@link StringValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, String value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link StringValue}.
+     *
+     * @param name name of the property
+     * @param first the first string in the list
+     * @param second the second string in the list
+     * @param others other strings in the list
+     */
+    public B set(String name, String first, String second, String... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (String other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link LongValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, long value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link LongValue}.
+     *
+     * @param name name of the property
+     * @param first the first long in the list
+     * @param second the second long in the list
+     * @param others other longs in the list
+     */
+    public B set(String name, long first, long second, long... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (long other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link DoubleValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, double value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link DoubleValue}.
+     *
+     * @param name name of the property
+     * @param first the first double in the list
+     * @param second the second double in the list
+     * @param others other doubles in the list
+     */
+    public B set(String name, double first, double second, double... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (double other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link BooleanValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, boolean value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link BooleanValue}.
+     *
+     * @param name name of the property
+     * @param first the first boolean in the list
+     * @param second the second boolean in the list
+     * @param others other booleans in the list
+     */
+    public B set(String name, boolean first, boolean second, boolean... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (boolean other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link DateTimeValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, DateTime value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link DateTimeValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link DateTime} in the list
+     * @param second the second {@link DateTime} in the list
+     * @param others other {@link DateTime}s in the list
+     */
+    public B set(String name, DateTime first, DateTime second, DateTime... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (DateTime other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link KeyValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Key value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link KeyValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link Key} in the list
+     * @param second the second {@link Key} in the list
+     * @param others other {@link Key}s in the list
+     */
+    public B set(String name, Key first, Key second, Key... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (Key other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link EntityValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, FullEntity value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link EntityValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link FullEntity} in the list
+     * @param second the second {@link FullEntity} in the list
+     * @param others other entities in the list
+     */
+    public B set(String name, FullEntity first, FullEntity second, FullEntity... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (FullEntity other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@link ListValue}.
+     *
+     * @param name name of the property
+     * @param values list of values associated with the property
+     */
     public B set(String name, List> values) {
       properties.put(name, of(values));
       return self();
     }
 
-    public B set(String name, Value value, Value... other) {
-      properties.put(name, of(value, other));
+    /**
+     * Sets a property of type {@link ListValue}.
+     *
+     * @param name name of the property
+     * @param first the first value in the list
+     * @param second the second value in the list
+     * @param others other values in the list
+     */
+    public B set(String name, Value first, Value second, Value... others) {
+      properties.put(name, ListValue.builder().addValue(first).addValue(second, others).build());
       return self();
     }
 
+    /**
+     * Sets a property of type {@link BlobValue}.
+     *
+     * @param name name of the property
+     * @param value value associated with the property
+     */
     public B set(String name, Blob value) {
       properties.put(name, of(value));
       return self();
     }
 
+    /**
+     * Sets a list property containing elements of type {@link BlobValue}.
+     *
+     * @param name name of the property
+     * @param first the first {@link Blob} in the list
+     * @param second the second {@link Blob} in the list
+     * @param others other {@link Blob}s in the list
+     */
+    public B set(String name, Blob first, Blob second, Blob... others) {
+      List values = new LinkedList<>();
+      values.add(of(first));
+      values.add(of(second));
+      for (Blob other : others) {
+        values.add(of(other));
+      }
+      properties.put(name, of(values));
+      return self();
+    }
+
+    /**
+     * Sets a property of type {@code NullValue}.
+     *
+     * @param name name of the property
+     */
     public B setNull(String name) {
       properties.put(name, of());
       return self();
@@ -348,8 +574,8 @@ public  FullEntity getEntity(String name) {
    * @throws ClassCastException if value is not a list of values
    */
   @SuppressWarnings("unchecked")
-  public List> getList(String name) {
-    return ((Value>>) getValue(name)).get();
+  public > List getList(String name) {
+    return (List) getValue(name).get();
   }
 
   /**
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
index a8ad7d4e7734..4ab6f51b6767 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/BaseKey.java
@@ -157,6 +157,8 @@ public String kind() {
     return leaf().kind();
   }
 
+  abstract BaseKey parent();
+
   @Override
   public int hashCode() {
     return Objects.hash(projectId(), namespace(), path());
diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
index 75a5d1381403..5306a685195a 100644
--- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
+++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Batch.java
@@ -24,14 +24,14 @@
  * to the Datastore upon {@link #submit}.
  * A usage example:
  * 
 {@code
- *   Entity entity1 = datastore.get(key1);
- *   Batch batch = datastore.newBatch();
- *   Entity entity2 = Entity.builder(key2).set("name", "John").build();
- *   entity1 = Entity.builder(entity1).clear().setNull("bla").build();
- *   Entity entity3 = Entity.builder(key3).set("title", "title").build();
- *   batch.update(entity1);
- *   batch.add(entity2, entity3);
- *   batch.submit();
+ * Entity entity1 = datastore.get(key1);
+ * Batch batch = datastore.newBatch();
+ * Entity entity2 = Entity.builder(key2).set("name", "John").build();
+ * entity1 = Entity.builder(entity1).clear().setNull("bla").build();
+ * Entity entity3 = Entity.builder(key3).set("title", "title").build();
+ * batch.update(entity1);
+ * batch.add(entity2, entity3);
+ * batch.submit();
  * } 
*/ public interface Batch extends DatastoreBatchWriter { diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java index e6ae166dbf07..7c03b69d9f39 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/GqlQuery.java @@ -43,27 +43,27 @@ *

A usage example:

* *

When the type of the results is known the preferred usage would be: - *

{@code
- *   Query query =
- *       Query.gqlQueryBuilder(Query.ResultType.ENTITY, "select * from kind").build();
- *   QueryResults results = datastore.run(query);
- *   while (results.hasNext()) {
- *     Entity entity = results.next();
- *     ...
- *   }
+ * 
 {@code
+ * Query query =
+ *     Query.gqlQueryBuilder(Query.ResultType.ENTITY, "select * from kind").build();
+ * QueryResults results = datastore.run(query);
+ * while (results.hasNext()) {
+ *   Entity entity = results.next();
+ *   ...
+ * }
  * } 
* *

When the type of the results is unknown you can use this approach: - *

{@code
- *   Query query = Query.gqlQueryBuilder("select __key__ from kind").build();
- *   QueryResults results = datastore.run(query);
- *   if (Key.class.isAssignableFrom(results.resultClass())) {
- *     QueryResults keys = (QueryResults) results;
- *     while (keys.hasNext()) {
- *       Key key = keys.next();
- *       ...
- *     }
+ * 
 {@code
+ * Query query = Query.gqlQueryBuilder("select __key__ from kind").build();
+ * QueryResults results = datastore.run(query);
+ * if (Key.class.isAssignableFrom(results.resultClass())) {
+ *   QueryResults keys = (QueryResults) results;
+ *   while (keys.hasNext()) {
+ *     Key key = keys.next();
+ *     ...
  *   }
+ * }
  * } 
* * @param the type of the result values this query will produce diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java index 2ccd59e725a8..2192384ef70b 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/IncompleteKey.java @@ -84,6 +84,29 @@ static IncompleteKey fromPb(DatastoreV1.Key keyPb) { return new IncompleteKey(projectId, namespace, path); } + /** + * Returns the key's parent. + */ + @Override + public Key parent() { + List ancestors = ancestors(); + if (ancestors.isEmpty()) { + return null; + } + PathElement parent = ancestors.get(ancestors.size() - 1); + Key.Builder keyBuilder; + if (parent.hasName()) { + keyBuilder = Key.builder(projectId(), parent.kind(), parent.name()); + } else { + keyBuilder = Key.builder(projectId(), parent.kind(), parent.id()); + } + String namespace = namespace(); + if (namespace != null) { + keyBuilder.namespace(namespace); + } + return keyBuilder.ancestors(ancestors.subList(0, ancestors.size() - 1)).build(); + } + public static Builder builder(String projectId, String kind) { return new Builder(projectId, kind); } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java index 41c7e82788b5..06282a2c79d1 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/ListValue.java @@ -70,17 +70,16 @@ private Builder() { super(ValueType.LIST); } - public Builder addValue(Value value) { + private void addValueHelper(Value value) { // see datastore_v1.proto definition for list_value Preconditions.checkArgument(value.type() != ValueType.LIST, "Cannot contain another list"); listBuilder.add(value); - return this; } public Builder addValue(Value first, Value... other) { - addValue(first); + addValueHelper(first); for (Value value : other) { - addValue(value); + addValueHelper(value); } return this; } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java index 15cca241e250..5892268f859c 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/StructuredQuery.java @@ -46,35 +46,35 @@ *

A usage example:

* *

A simple query that returns all entities for a specific kind - *

{@code
- *   Query query = Query.entityQueryBuilder().kind(kind).build();
- *   QueryResults results = datastore.run(query);
- *   while (results.hasNext()) {
- *     Entity entity = results.next();
- *     ...
- *   }
+ * 
 {@code
+ * Query query = Query.entityQueryBuilder().kind(kind).build();
+ * QueryResults results = datastore.run(query);
+ * while (results.hasNext()) {
+ *   Entity entity = results.next();
+ *   ...
+ * }
  * } 
* *

A simple key-only query of all entities for a specific kind - *

{@code
- *   Query keyOnlyQuery =  Query.keyQueryBuilder().kind(KIND1).build();
- *   QueryResults results = datastore.run(keyOnlyQuery);
- *   ...
+ * 
 {@code
+ * Query keyOnlyQuery =  Query.keyQueryBuilder().kind(KIND1).build();
+ * QueryResults results = datastore.run(keyOnlyQuery);
+ * ...
  * } 
* *

A less trivial example of a projection query that returns the first 10 results * of "age" and "name" properties (sorted and grouped by "age") with an age greater than 18 - *

{@code
- *   Query query = Query.projectionEntityQueryBuilder()
- *       .kind(kind)
- *       .projection(Projection.property("age"), Projection.first("name"))
- *       .filter(PropertyFilter.gt("age", 18))
- *       .groupBy("age")
- *       .orderBy(OrderBy.asc("age"))
- *       .limit(10)
- *       .build();
- *   QueryResults results = datastore.run(query);
- *   ...
+ * 
 {@code
+ * Query query = Query.projectionEntityQueryBuilder()
+ *     .kind(kind)
+ *     .projection(Projection.property("age"), Projection.first("name"))
+ *     .filter(PropertyFilter.gt("age", 18))
+ *     .groupBy("age")
+ *     .orderBy(OrderBy.asc("age"))
+ *     .limit(10)
+ *     .build();
+ * QueryResults results = datastore.run(query);
+ * ...
  * } 
* * @param the type of the result values this query will produce diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java index 8089c0130f5d..78ee217f31e7 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/Transaction.java @@ -30,21 +30,21 @@ * the Datastore upon {@code commit}. * A usage example: *
 {@code
- *   Transaction transaction = datastore.newTransaction();
- *   try {
- *     Entity entity = transaction.get(key);
- *     if (!entity.contains("last_name") || entity.isNull("last_name")) {
- *       String[] name = entity.getString("name").split(" ");
- *       entity = Entity.builder(entity).remove("name").set("first_name", name[0])
- *           .set("last_name", name[1]).build();
- *       transaction.update(entity);
- *       transaction.commit();
- *     }
- *   } finally {
- *     if (transaction.active()) {
- *       transaction.rollback();
- *     }
+ * Transaction transaction = datastore.newTransaction();
+ * try {
+ *   Entity entity = transaction.get(key);
+ *   if (!entity.contains("last_name") || entity.isNull("last_name")) {
+ *     String[] name = entity.getString("name").split(" ");
+ *     entity = Entity.builder(entity).remove("name").set("first_name", name[0])
+ *         .set("last_name", name[1]).build();
+ *     transaction.update(entity);
+ *     transaction.commit();
  *   }
+ * } finally {
+ *   if (transaction.active()) {
+ *     transaction.rollback();
+ *   }
+ * }
  * } 
* * @see diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java index 1404b2817802..fbf468d6458d 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/package-info.java @@ -17,35 +17,38 @@ /** * A client to the Google Cloud Datastore. * - *

Here's a simple usage example for using gcloud-java from App/Compute Engine: + *

Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * shows how to create a Datastore entity. For the complete source code see + * + * CreateEntity.java. *

 {@code
  * Datastore datastore = DatastoreOptions.defaultInstance().service();
- * KeyFactory keyFactory = datastore.newKeyFactory().kind(kind);
- * Key key = keyFactory.newKey(keyName);
+ * KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind");
+ * Key key = keyFactory.newKey("keyName");
+ * Entity entity = Entity.builder(key)
+ *     .set("name", "John Doe")
+ *     .set("age", 30)
+ *     .set("access_time", DateTime.now())
+ *     .build();
+ * datastore.put(entity);
+ * } 
+ *

+ * This second example shows how to get and update a Datastore entity if it exists. For the complete + * source code see + * + * UpdateEntity.java. + *

 {@code
+ * Datastore datastore = DatastoreOptions.defaultInstance().service();
+ * KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind");
+ * Key key = keyFactory.newKey("keyName");
  * Entity entity = datastore.get(key);
- * if (entity == null) {
- *   entity = Entity.builder(key)
- *       .set("name", "John Do")
- *       .set("age", LongValue.builder(100).indexed(false).build())
- *       .set("updated", false)
+ * if (entity != null) {
+ *   System.out.println("Updating access_time for " + entity.getString("name"));
+ *   entity = Entity.builder(entity)
+ *       .set("access_time", DateTime.now())
  *       .build();
- *   datastore.put(entity);
- * } else {
- *   boolean updated = entity.getBoolean("updated");
- *   if (!updated) {
- *     String[] name = entity.getString("name").split(" ");
- *     entity = Entity.builder(entity)
- *         .set("name", name[0])
- *         .set("last_name", StringValue.builder(name[1]).indexed(false).build())
- *         .set("updated", true)
- *         .remove("old_property")
- *         .set("new_property", 1.1)
- *         .build();
- *     datastore.update(entity);
- *   }
- * }
- * } 
- * + * datastore.update(entity); + * }}
*

When using gcloud-java from outside of App/Compute Engine, you have to specify a * project ID and diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java index 7c60da50b0b0..fdb6774f810f 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/datastore/testing/LocalGcdHelper.java @@ -17,6 +17,7 @@ package com.google.gcloud.datastore.testing; import static com.google.common.base.MoreObjects.firstNonNull; +import static com.google.common.base.Preconditions.checkArgument; import static java.nio.charset.StandardCharsets.UTF_8; import com.google.common.base.Strings; @@ -85,6 +86,7 @@ public class LocalGcdHelper { private static final String GCLOUD = "gcloud"; private static final Path INSTALLED_GCD_PATH; private static final String GCD_VERSION_PREFIX = "gcd-emulator "; + private static final double DEFAULT_CONSISTENCY = 0.9; static { INSTALLED_GCD_PATH = installedGcdPath(); @@ -398,9 +400,15 @@ public LocalGcdHelper(String projectId, int port) { * All content is written to a temporary directory that will be deleted when * {@link #stop()} is called or when the program terminates) to make sure that no left-over * data from prior runs is used. + * + * @param consistency the fraction of job application attempts that will succeed, with 0.0 + * resulting in no attempts succeeding, and 1.0 resulting in all attempts succeeding. Defaults + * to 0.9. Note that setting this to 1.0 may mask incorrect assumptions about the consistency + * of non-ancestor queries; non-ancestor queries are eventually consistent. */ - public void start() throws IOException, InterruptedException { + public void start(double consistency) throws IOException, InterruptedException { // send a quick request in case we have a hanging process from a previous run + checkArgument(consistency >= 0.0 && consistency <= 1.0, "Consistency must be between 0 and 1"); sendQuitRequest(port); // Each run is associated with its own folder that is deleted once test completes. gcdPath = Files.createTempDirectory("gcd"); @@ -415,7 +423,7 @@ public void start() throws IOException, InterruptedException { } else { gcdExecutablePath = INSTALLED_GCD_PATH; } - startGcd(gcdExecutablePath); + startGcd(gcdExecutablePath, consistency); } private void downloadGcd() throws IOException { @@ -453,7 +461,8 @@ private void downloadGcd() throws IOException { } } - private void startGcd(Path executablePath) throws IOException, InterruptedException { + private void startGcd(Path executablePath, double consistency) + throws IOException, InterruptedException { // cleanup any possible data for the same project File datasetFolder = new File(gcdPath.toFile(), projectId); deleteRecurse(datasetFolder.toPath()); @@ -486,7 +495,8 @@ private void startGcd(Path executablePath) throws IOException, InterruptedExcept startProcess = CommandWrapper.create() .command(gcdAbsolutePath.toString(), "start", "--testing", "--allow_remote_shutdown", - "--port=" + Integer.toString(port), projectId) + "--port=" + Integer.toString(port), "--consistency=" + Double.toString(consistency), + projectId) .directory(gcdPath) .start(); processReader = ProcessStreamReader.start(startProcess.getInputStream()); @@ -526,6 +536,7 @@ private static void extractFile(ZipInputStream zipIn, File filePath) throws IOEx public static boolean sendQuitRequest(int port) { StringBuilder result = new StringBuilder(); + String shutdownMsg = "Shutting down local server"; try { URL url = new URL("http", "localhost", port, "/_ah/admin/quit"); HttpURLConnection con = (HttpURLConnection) url.openConnection(); @@ -537,13 +548,13 @@ public static boolean sendQuitRequest(int port) { out.flush(); InputStream in = con.getInputStream(); int currByte = 0; - while ((currByte = in.read()) != -1) { + while ((currByte = in.read()) != -1 && result.length() < shutdownMsg.length()) { result.append(((char) currByte)); } } catch (IOException ignore) { // ignore } - return result.toString().startsWith("Shutting down local server"); + return result.toString().startsWith(shutdownMsg); } public void stop() throws IOException, InterruptedException { @@ -578,10 +589,10 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IO }); } - public static LocalGcdHelper start(String projectId, int port) + public static LocalGcdHelper start(String projectId, int port, double consistency) throws IOException, InterruptedException { LocalGcdHelper helper = new LocalGcdHelper(projectId, port); - helper.start(); + helper.start(consistency); return helper; } @@ -593,7 +604,9 @@ public static void main(String... args) throws IOException, InterruptedException switch (action) { case "START": if (!isActive(DEFAULT_PROJECT_ID, port)) { - LocalGcdHelper helper = start(DEFAULT_PROJECT_ID, port); + double consistency = parsedArgs.get("consistency") == null + ? DEFAULT_CONSISTENCY : Double.parseDouble(parsedArgs.get("consistency")); + LocalGcdHelper helper = start(DEFAULT_PROJECT_ID, port, consistency); try (FileWriter writer = new FileWriter(".local_gcd_helper")) { writer.write(helper.gcdPath.toAbsolutePath().toString() + System.lineSeparator()); writer.write(Integer.toString(port)); diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DatastoreRpc.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DatastoreRpc.java index fd916e0a1c87..4d4540c70196 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DatastoreRpc.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DatastoreRpc.java @@ -35,16 +35,46 @@ */ public interface DatastoreRpc { - AllocateIdsResponse allocateIds(AllocateIdsRequest request) throws DatastoreException; + /** + * Sends an allocate IDs request. + * + * @throws DatastoreException upon failure + */ + AllocateIdsResponse allocateIds(AllocateIdsRequest request); + /** + * Sends a begin transaction request. + * + * @throws DatastoreException upon failure + */ BeginTransactionResponse beginTransaction(BeginTransactionRequest request) throws DatastoreException; - CommitResponse commit(CommitRequest request) throws DatastoreException; + /** + * Sends a commit request. + * + * @throws DatastoreException upon failure + */ + CommitResponse commit(CommitRequest request); - LookupResponse lookup(LookupRequest request) throws DatastoreException; + /** + * Sends a lookup request. + * + * @throws DatastoreException upon failure + */ + LookupResponse lookup(LookupRequest request); - RollbackResponse rollback(RollbackRequest request) throws DatastoreException; + /** + * Sends a rollback request. + * + * @throws DatastoreException upon failure + */ + RollbackResponse rollback(RollbackRequest request); - RunQueryResponse runQuery(RunQueryRequest request) throws DatastoreException; + /** + * Sends a request to run a query. + * + * @throws DatastoreException upon failure + */ + RunQueryResponse runQuery(RunQueryRequest request); } diff --git a/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DefaultDatastoreRpc.java b/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DefaultDatastoreRpc.java index c82ff9689f68..f679cc0d5826 100644 --- a/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DefaultDatastoreRpc.java +++ b/gcloud-java-datastore/src/main/java/com/google/gcloud/spi/DefaultDatastoreRpc.java @@ -111,8 +111,7 @@ private static DatastoreException translate( } @Override - public AllocateIdsResponse allocateIds(AllocateIdsRequest request) - throws DatastoreException { + public AllocateIdsResponse allocateIds(AllocateIdsRequest request) { try { return client.allocateIds(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { @@ -121,8 +120,7 @@ public AllocateIdsResponse allocateIds(AllocateIdsRequest request) } @Override - public BeginTransactionResponse beginTransaction(BeginTransactionRequest request) - throws DatastoreException { + public BeginTransactionResponse beginTransaction(BeginTransactionRequest request) { try { return client.beginTransaction(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { @@ -131,7 +129,7 @@ public BeginTransactionResponse beginTransaction(BeginTransactionRequest request } @Override - public CommitResponse commit(CommitRequest request) throws DatastoreException { + public CommitResponse commit(CommitRequest request) { try { return client.commit(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { @@ -140,7 +138,7 @@ public CommitResponse commit(CommitRequest request) throws DatastoreException { } @Override - public LookupResponse lookup(LookupRequest request) throws DatastoreException { + public LookupResponse lookup(LookupRequest request) { try { return client.lookup(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { @@ -149,7 +147,7 @@ public LookupResponse lookup(LookupRequest request) throws DatastoreException { } @Override - public RollbackResponse rollback(RollbackRequest request) throws DatastoreException { + public RollbackResponse rollback(RollbackRequest request) { try { return client.rollback(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { @@ -158,7 +156,7 @@ public RollbackResponse rollback(RollbackRequest request) throws DatastoreExcept } @Override - public RunQueryResponse runQuery(RunQueryRequest request) throws DatastoreException { + public RunQueryResponse runQuery(RunQueryRequest request) { try { return client.runQuery(request); } catch (com.google.api.services.datastore.client.DatastoreException ex) { diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java index 5ece01508d3a..a69ea5e20e3b 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseEntityTest.java @@ -67,6 +67,16 @@ public void setUp() { builder.set("list1", NullValue.of(), StringValue.of("foo")); builder.set("list2", ImmutableList.of(LongValue.of(10), DoubleValue.of(2))); builder.set("list3", Collections.singletonList(BooleanValue.of(true))); + builder.set( + "blobList", BLOB, Blob.copyFrom(new byte[] {3, 4}), Blob.copyFrom(new byte[] {5, 6})); + builder.set("booleanList", true, false, true); + builder.set("dateTimeList", DateTime.now(), DateTime.now(), DateTime.now()); + builder.set("doubleList", 12.3, 4.56, .789); + builder.set("keyList", KEY, Key.builder("ds2", "k2", "n2").build(), + Key.builder("ds3", "k3", "n3").build()); + builder.set("entityList", ENTITY, PARTIAL_ENTITY); + builder.set("stringList", "s1", "s2", "s3"); + builder.set("longList", 1, 23, 456); } @Test @@ -183,6 +193,17 @@ public void testGetList() throws Exception { assertEquals(Boolean.TRUE, list.get(0).get()); entity = builder.set("list1", ListValue.of(list)).build(); assertEquals(list, entity.getList("list1")); + List> stringList = entity.getList("stringList"); + assertEquals( + ImmutableList.of(StringValue.of("s1"), StringValue.of("s2"), StringValue.of("s3")), + stringList); + List> doubleList = entity.getList("doubleList"); + assertEquals( + ImmutableList.of(DoubleValue.of(12.3), DoubleValue.of(4.56), DoubleValue.of(.789)), + doubleList); + List entityList = entity.getList("entityList"); + assertEquals( + ImmutableList.of(EntityValue.of(ENTITY), EntityValue.of(PARTIAL_ENTITY)), entityList); } @Test @@ -198,7 +219,9 @@ public void testGetBlob() throws Exception { public void testNames() throws Exception { Set names = ImmutableSet.builder() .add("string", "stringValue", "boolean", "double", "long", "list1", "list2", "list3") - .add("entity", "partialEntity", "null", "dateTime", "blob", "key") + .add("entity", "partialEntity", "null", "dateTime", "blob", "key", "blobList") + .add("booleanList", "dateTimeList", "doubleList", "keyList", "entityList", "stringList") + .add("longList") .build(); BaseEntity entity = builder.build(); assertEquals(names, entity.names()); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java index 8615ee025bd1..43db4695b191 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/BaseKeyTest.java @@ -50,6 +50,11 @@ protected BaseKey build() { protected Object fromPb(byte[] bytesPb) throws InvalidProtocolBufferException { return null; } + + @Override + protected BaseKey parent() { + return null; + } }; } } diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java index 4d62224172f9..f7bdcb89bcec 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreExceptionTest.java @@ -78,7 +78,8 @@ public void testDatastoreException() throws Exception { @Test public void testTranslateAndThrow() throws Exception { DatastoreException cause = new DatastoreException(503, "message", "UNAVAILABLE"); - RetryHelper.RetryHelperException exceptionMock = createMock(RetryHelper.RetryHelperException.class); + RetryHelper.RetryHelperException exceptionMock = + createMock(RetryHelper.RetryHelperException.class); expect(exceptionMock.getCause()).andReturn(cause).times(2); replay(exceptionMock); try { diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java index e9eed027e8e0..5d106fc7d31d 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/DatastoreTest.java @@ -31,6 +31,8 @@ import com.google.api.services.datastore.DatastoreV1.RunQueryRequest; import com.google.api.services.datastore.DatastoreV1.RunQueryResponse; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; +import com.google.gcloud.AuthCredentials; import com.google.gcloud.RetryParams; import com.google.gcloud.datastore.Query.ResultType; import com.google.gcloud.datastore.StructuredQuery.OrderBy; @@ -89,8 +91,8 @@ public class DatastoreTest { FullEntity.builder(INCOMPLETE_KEY2).set("str", STR_VALUE).set("bool", BOOL_VALUE) .set("list", LIST_VALUE1).build(); private static final FullEntity PARTIAL_ENTITY2 = - FullEntity.builder(PARTIAL_ENTITY1).remove("str").set("bool", true). - set("list", LIST_VALUE1.get()).build(); + FullEntity.builder(PARTIAL_ENTITY1).remove("str").set("bool", true) + .set("list", LIST_VALUE1.get()).build(); private static final FullEntity PARTIAL_ENTITY3 = FullEntity.builder(PARTIAL_ENTITY1).key(IncompleteKey.builder(PROJECT_ID, KIND3).build()) .build(); @@ -118,7 +120,7 @@ public class DatastoreTest { @BeforeClass public static void beforeClass() throws IOException, InterruptedException { if (!LocalGcdHelper.isActive(PROJECT_ID, PORT)) { - gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 1.0); } } @@ -127,6 +129,7 @@ public void setUp() { options = DatastoreOptions.builder() .projectId(PROJECT_ID) .host("http://localhost:" + PORT) + .authCredentials(AuthCredentials.noAuth()) .retryParams(RetryParams.noRetries()) .build(); datastore = options.service(); @@ -471,9 +474,13 @@ public void testQueryPaginationWithLimit() throws DatastoreException { EasyMock.expect(rpcFactoryMock.create(EasyMock.anyObject(DatastoreOptions.class))) .andReturn(rpcMock); List responses = buildResponsesForQueryPaginationWithLimit(); - for (int i = 0; i < responses.size(); i++) { + List endCursors = Lists.newArrayListWithCapacity(responses.size()); + for (RunQueryResponse response : responses) { EasyMock.expect(rpcMock.runQuery(EasyMock.anyObject(RunQueryRequest.class))) - .andReturn(responses.get(i)); + .andReturn(response); + if (response.getBatch().getMoreResults() != QueryResultBatch.MoreResultsType.NOT_FINISHED) { + endCursors.add(response.getBatch().getEndCursor()); + } } EasyMock.replay(rpcFactoryMock, rpcMock); Datastore mockDatastore = options.toBuilder() @@ -483,6 +490,7 @@ public void testQueryPaginationWithLimit() throws DatastoreException { .service(); int limit = 2; int totalCount = 0; + Iterator cursorIter = endCursors.iterator(); StructuredQuery query = Query.entityQueryBuilder().limit(limit).build(); while (true) { QueryResults results = mockDatastore.run(query); @@ -492,6 +500,9 @@ public void testQueryPaginationWithLimit() throws DatastoreException { resultCount++; totalCount++; } + assertTrue(cursorIter.hasNext()); + Cursor expectedEndCursor = Cursor.copyFrom(cursorIter.next().toByteArray()); + assertEquals(expectedEndCursor, results.cursorAfter()); if (resultCount < limit) { break; } @@ -505,19 +516,20 @@ private List buildResponsesForQueryPaginationWithLimit() { Entity entity4 = Entity.builder(KEY4).set("value", StringValue.of("value")).build(); Entity entity5 = Entity.builder(KEY5).set("value", "value").build(); datastore.add(ENTITY3, entity4, entity5); + DatastoreRpc datastoreRpc = datastore.options().rpc(); List responses = new ArrayList<>(); Query query = Query.entityQueryBuilder().build(); RunQueryRequest.Builder requestPb = RunQueryRequest.newBuilder(); query.populatePb(requestPb); QueryResultBatch queryResultBatchPb = RunQueryResponse.newBuilder() - .mergeFrom(((DatastoreImpl) datastore).runQuery(requestPb.build())) + .mergeFrom(datastoreRpc.runQuery(requestPb.build())) .getBatch(); QueryResultBatch queryResultBatchPb1 = QueryResultBatch.newBuilder() .mergeFrom(queryResultBatchPb) .setMoreResults(QueryResultBatch.MoreResultsType.NOT_FINISHED) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(0, 1)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(0).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("a")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb1).build()); QueryResultBatch queryResultBatchPb2 = QueryResultBatch.newBuilder() @@ -534,7 +546,7 @@ private List buildResponsesForQueryPaginationWithLimit() { .setMoreResults(QueryResultBatch.MoreResultsType.MORE_RESULTS_AFTER_LIMIT) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(2, 4)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(3).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("b")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb3).build()); QueryResultBatch queryResultBatchPb4 = QueryResultBatch.newBuilder() @@ -542,7 +554,7 @@ private List buildResponsesForQueryPaginationWithLimit() { .setMoreResults(QueryResultBatch.MoreResultsType.NO_MORE_RESULTS) .clearEntityResult() .addAllEntityResult(queryResultBatchPb.getEntityResultList().subList(4, 5)) - .setEndCursor(queryResultBatchPb.getEntityResultList().get(4).getCursor()) + .setEndCursor(ByteString.copyFromUtf8("c")) .build(); responses.add(RunQueryResponse.newBuilder().setBatch(queryResultBatchPb4).build()); return responses; diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java index 7edbf133d330..acd1dfd3c9e3 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/IncompleteKeyTest.java @@ -17,29 +17,47 @@ package com.google.gcloud.datastore; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; +import org.junit.Before; import org.junit.Test; public class IncompleteKeyTest { + private static IncompleteKey pk1, pk2; + private static Key parent1; + + @Before + public void setUp() { + pk1 = IncompleteKey.builder("ds", "kind1").build(); + parent1 = Key.builder("ds", "kind2", 10).namespace("ns").build(); + pk2 = IncompleteKey.builder(parent1, "kind3").build(); + } + @Test public void testBuilders() throws Exception { - IncompleteKey pk1 = IncompleteKey.builder("ds", "kind1").build(); assertEquals("ds", pk1.projectId()); assertEquals("kind1", pk1.kind()); assertTrue(pk1.ancestors().isEmpty()); - Key parent = Key.builder("ds", "kind2", 10).build(); - IncompleteKey pk2 = IncompleteKey.builder(parent, "kind3").build(); assertEquals("ds", pk2.projectId()); assertEquals("kind3", pk2.kind()); - assertEquals(parent.path(), pk2.ancestors()); + assertEquals(parent1.path(), pk2.ancestors()); assertEquals(pk2, IncompleteKey.builder(pk2).build()); IncompleteKey pk3 = IncompleteKey.builder(pk2).kind("kind4").build(); assertEquals("ds", pk3.projectId()); assertEquals("kind4", pk3.kind()); - assertEquals(parent.path(), pk3.ancestors()); + assertEquals(parent1.path(), pk3.ancestors()); + } + + @Test + public void testParent() { + assertNull(pk1.parent()); + assertEquals(parent1, pk2.parent()); + Key parent2 = Key.builder("ds", "kind3", "name").namespace("ns").build(); + IncompleteKey pk3 = IncompleteKey.builder(parent2, "kind3").build(); + assertEquals(parent2, pk3.parent()); } } diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java index 40ea62c5a7e0..5d761a713506 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/LocalGcdHelperTest.java @@ -49,7 +49,7 @@ public void testFindAvailablePort() { @Test public void testSendQuitRequest() throws IOException, InterruptedException { - LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 0.75); assertTrue(LocalGcdHelper.sendQuitRequest(PORT)); long timeoutMillis = 30000; long startTime = System.currentTimeMillis(); @@ -64,7 +64,7 @@ public void testSendQuitRequest() throws IOException, InterruptedException { @Test public void testStartStop() throws IOException, InterruptedException { - LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT); + LocalGcdHelper gcdHelper = LocalGcdHelper.start(PROJECT_ID, PORT, 0.75); assertFalse(LocalGcdHelper.isActive("wrong-project-id", PORT)); assertTrue(LocalGcdHelper.isActive(PROJECT_ID, PORT)); gcdHelper.stop(); diff --git a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java index b0d188cae16e..4b6589efd723 100644 --- a/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java +++ b/gcloud-java-datastore/src/test/java/com/google/gcloud/datastore/StructuredQueryTest.java @@ -40,7 +40,8 @@ public class StructuredQueryTest { private static final Cursor END_CURSOR = Cursor.copyFrom(new byte[] {10}); private static final int OFFSET = 42; private static final Integer LIMIT = 43; - private static final Filter FILTER = CompositeFilter.and(PropertyFilter.gt("p1", 10), PropertyFilter.eq("a", "v")); + private static final Filter FILTER = + CompositeFilter.and(PropertyFilter.gt("p1", 10), PropertyFilter.eq("a", "v")); private static final OrderBy ORDER_BY_1 = OrderBy.asc("p2"); private static final OrderBy ORDER_BY_2 = OrderBy.desc("p3"); private static final List ORDER_BY = ImmutableList.of(ORDER_BY_1, ORDER_BY_2); diff --git a/gcloud-java-examples/README.md b/gcloud-java-examples/README.md index 8030d14d09e7..7d54b8f3e1a9 100644 --- a/gcloud-java-examples/README.md +++ b/gcloud-java-examples/README.md @@ -6,6 +6,8 @@ Examples for gcloud-java (Java idiomatic client for [Google Cloud Platform][clou [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-examples.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-examples.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [Examples] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/examples/package-summary.html) @@ -17,16 +19,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-examples - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-examples:0.1.3' +compile 'com.google.gcloud:gcloud-java-examples:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-examples" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java-examples" % "0.1.5" ``` To run examples from your command line: @@ -53,39 +55,39 @@ To run examples from your command line: ``` Then you are ready to run the following example: ``` - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create dataset new_dataset_id" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="list tables new_dataset_id" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create dataset new_dataset_id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="create table new_dataset_id new_table_id field_name:string" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="list tables new_dataset_id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="load new_dataset_id new_table_id CSV gs://my_bucket/my_csv_file" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample" -Dexec.args="query 'select * from new_dataset_id.new_table_id'" ``` * Here's an example run of `DatastoreExample`. Be sure to change the placeholder project ID "your-project-id" with your own project ID. Also note that you have to enable the Google Cloud Datastore API on the [Google Developers Console][developers-console] before running the following commands. ``` - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name display" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" -Dexec.args="your-project-id my_name delete" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name add my\ comment" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name display" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" -Dexec.args="your-project-id my_name delete" ``` * Here's an example run of `ResourceManagerExample`. Be sure to change the placeholder project ID "your-project-id" with your own globally unique project ID. ``` - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="create your-project-id" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="list" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" -Dexec.args="get your-project-id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="create your-project-id" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="list" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" -Dexec.args="get your-project-id" ``` * Here's an example run of `StorageExample`. Before running the example, go to the [Google Developers Console][developers-console] to ensure that Google Cloud Storage API is enabled and that you have a bucket. Also ensure that you have a test file (`test.txt` is chosen here) to upload to Cloud Storage stored locally on your machine. ``` - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="upload /path/to/test.txt " - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="list " - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="download test.txt" - mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample" -Dexec.args="delete test.txt" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="upload /path/to/test.txt " + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="list " + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="download test.txt" + mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample" -Dexec.args="delete test.txt" ``` Troubleshooting diff --git a/gcloud-java-examples/pom.xml b/gcloud-java-examples/pom.xml index 5597f1f44132..111308658c2e 100644 --- a/gcloud-java-examples/pom.xml +++ b/gcloud-java-examples/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-examples jar GCloud Java examples @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT gcloud-java-examples diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java similarity index 97% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java index 1b1478eb5be5..c8fbe7289f9c 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/BigQueryExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/BigQueryExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.bigquery; import com.google.common.collect.ImmutableMap; import com.google.gcloud.WriteChannel; @@ -22,6 +22,7 @@ import com.google.gcloud.bigquery.BigQueryError; import com.google.gcloud.bigquery.BigQueryOptions; import com.google.gcloud.bigquery.CopyJobConfiguration; +import com.google.gcloud.bigquery.Dataset; import com.google.gcloud.bigquery.DatasetId; import com.google.gcloud.bigquery.DatasetInfo; import com.google.gcloud.bigquery.ExternalTableDefinition; @@ -29,14 +30,15 @@ import com.google.gcloud.bigquery.Field; import com.google.gcloud.bigquery.FieldValue; import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; import com.google.gcloud.bigquery.JobId; import com.google.gcloud.bigquery.JobInfo; -import com.google.gcloud.bigquery.JobStatus; import com.google.gcloud.bigquery.LoadJobConfiguration; import com.google.gcloud.bigquery.QueryRequest; import com.google.gcloud.bigquery.QueryResponse; import com.google.gcloud.bigquery.Schema; import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; import com.google.gcloud.bigquery.TableId; import com.google.gcloud.bigquery.TableInfo; import com.google.gcloud.bigquery.ViewDefinition; @@ -61,7 +63,7 @@ *

  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - - *
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.BigQueryExample"
    + * 
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.bigquery.BigQueryExample"
      *  -Dexec.args="[]
      *  list datasets |
      *  list tables  |
    @@ -176,7 +178,7 @@ Void parse(String... args) throws Exception {
       private static class ListDatasetsAction extends NoArgsAction {
         @Override
         public void run(BigQuery bigquery, Void arg) {
    -      Iterator datasetInfoIterator = bigquery.listDatasets().iterateAll();
    +      Iterator datasetInfoIterator = bigquery.listDatasets().iterateAll();
           while (datasetInfoIterator.hasNext()) {
             System.out.println(datasetInfoIterator.next());
           }
    @@ -211,7 +213,7 @@ public String params() {
       private static class ListTablesAction extends DatasetAction {
         @Override
         public void run(BigQuery bigquery, DatasetId datasetId) {
    -      Iterator tableInfoIterator = bigquery.listTables(datasetId).iterateAll();
    +      Iterator
  • tableInfoIterator = bigquery.listTables(datasetId).iterateAll(); while (tableInfoIterator.hasNext()) { System.out.println(tableInfoIterator.next()); } @@ -355,7 +357,7 @@ public String params() { private static class ListJobsAction extends NoArgsAction { @Override public void run(BigQuery bigquery, Void arg) { - Iterator datasetInfoIterator = bigquery.listJobs().iterateAll(); + Iterator datasetInfoIterator = bigquery.listJobs().iterateAll(); while (datasetInfoIterator.hasNext()) { System.out.println(datasetInfoIterator.next()); } @@ -393,7 +395,7 @@ public void run(BigQuery bigquery, JobId jobId) { private abstract static class CreateTableAction extends BigQueryAction { @Override void run(BigQuery bigquery, TableInfo table) throws Exception { - TableInfo createTable = bigquery.create(table); + Table createTable = bigquery.create(table); System.out.println("Created table:"); System.out.println(createTable.toString()); } @@ -521,11 +523,10 @@ private abstract static class JobRunAction extends BigQueryAction { @Override void run(BigQuery bigquery, JobInfo job) throws Exception { System.out.println("Creating job"); - JobInfo startedJob = bigquery.create(job); - while (startedJob.status().state() != JobStatus.State.DONE) { + Job startedJob = bigquery.create(job); + while (!startedJob.isDone()) { System.out.println("Waiting for job " + startedJob.jobId().job() + " to complete"); Thread.sleep(1000L); - startedJob = bigquery.getJob(startedJob.jobId()); } if (startedJob.status().error() == null) { System.out.println("Job " + startedJob.jobId().job() + " succeeded"); diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java new file mode 100644 index 000000000000..857f6b43d013 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/CreateTableAndLoadData.java @@ -0,0 +1,64 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.bigquery.snippets; + +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQueryOptions; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FormatOptions; +import com.google.gcloud.bigquery.Job; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.Table; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; + +/** + * A snippet for Google Cloud BigQuery showing how to get a BigQuery table or create it if it does + * not exist. The snippet also starts a BigQuery job to load data into the table from a Cloud + * Storage blob and wait until the job completes. + */ +public class CreateTableAndLoadData { + + public static void main(String... args) throws InterruptedException { + BigQuery bigquery = BigQueryOptions.defaultInstance().service(); + TableId tableId = TableId.of("dataset", "table"); + Table table = bigquery.getTable(tableId); + if (table == null) { + System.out.println("Creating table " + tableId); + Field integerField = Field.of("fieldName", Field.Type.integer()); + Schema schema = Schema.of(integerField); + table = bigquery.create(TableInfo.of(tableId, StandardTableDefinition.of(schema))); + } + System.out.println("Loading data into table " + tableId); + Job loadJob = table.load(FormatOptions.csv(), "gs://bucket/path"); + while (!loadJob.isDone()) { + Thread.sleep(1000L); + } + if (loadJob.status().error() != null) { + System.out.println("Job completed with errors"); + } else { + System.out.println("Job succeeded"); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java new file mode 100644 index 000000000000..f421bc832441 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/bigquery/snippets/InsertDataAndQueryTable.java @@ -0,0 +1,102 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.bigquery.snippets; + +import com.google.gcloud.bigquery.BigQuery; +import com.google.gcloud.bigquery.BigQueryOptions; +import com.google.gcloud.bigquery.DatasetInfo; +import com.google.gcloud.bigquery.Field; +import com.google.gcloud.bigquery.FieldValue; +import com.google.gcloud.bigquery.InsertAllRequest; +import com.google.gcloud.bigquery.InsertAllResponse; +import com.google.gcloud.bigquery.QueryRequest; +import com.google.gcloud.bigquery.QueryResponse; +import com.google.gcloud.bigquery.Schema; +import com.google.gcloud.bigquery.StandardTableDefinition; +import com.google.gcloud.bigquery.TableId; +import com.google.gcloud.bigquery.TableInfo; + +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; + +/** + * A snippet for Google Cloud BigQuery showing how to create a BigQuery dataset and table. Once + * created, the snippet streams data into the table and then queries it. + */ +public class InsertDataAndQueryTable { + + public static void main(String... args) throws InterruptedException { + // Create a service instance + BigQuery bigquery = BigQueryOptions.defaultInstance().service(); + + // Create a dataset + String datasetId = "my_dataset_id"; + bigquery.create(DatasetInfo.builder(datasetId).build()); + + TableId tableId = TableId.of(datasetId, "my_table_id"); + // Table field definition + Field stringField = Field.of("StringField", Field.Type.string()); + // Table schema definition + Schema schema = Schema.of(stringField); + // Create a table + StandardTableDefinition tableDefinition = StandardTableDefinition.of(schema); + bigquery.create(TableInfo.of(tableId, tableDefinition)); + + // Define rows to insert + Map firstRow = new HashMap<>(); + Map secondRow = new HashMap<>(); + firstRow.put("StringField", "value1"); + secondRow.put("StringField", "value2"); + // Create an insert request + InsertAllRequest insertRequest = InsertAllRequest.builder(tableId) + .addRow(firstRow) + .addRow(secondRow) + .build(); + // Insert rows + InsertAllResponse insertResponse = bigquery.insertAll(insertRequest); + // Check if errors occurred + if (insertResponse.hasErrors()) { + System.out.println("Errors occurred while inserting rows"); + } + + // Create a query request + QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id") + .maxWaitTime(60000L) + .maxResults(1000L) + .build(); + // Request query to be executed and wait for results + QueryResponse queryResponse = bigquery.query(queryRequest); + while (!queryResponse.jobCompleted()) { + Thread.sleep(1000L); + queryResponse = bigquery.getQueryResults(queryResponse.jobId()); + } + // Read rows + Iterator> rowIterator = queryResponse.result().iterateAll(); + System.out.println("Table rows:"); + while (rowIterator.hasNext()) { + System.out.println(rowIterator.next()); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java similarity index 98% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java index 1e65a018a1fb..cc4331734200 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/DatastoreExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/DatastoreExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.datastore; import com.google.gcloud.datastore.Datastore; import com.google.gcloud.datastore.DatastoreOptions; @@ -44,7 +44,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - {@code mvn exec:java - * -Dexec.mainClass="com.google.gcloud.examples.DatastoreExample" + * -Dexec.mainClass="com.google.gcloud.examples.datastore.DatastoreExample" * -Dexec.args="[projectId] [user] [delete|display|add comment]"}
  • * */ @@ -67,7 +67,7 @@ private static class DeleteAction implements DatastoreAction { public void run(Transaction tx, Key userKey, String... args) { Entity user = tx.get(userKey); if (user == null) { - System.out.println("Nothing to delete, user does not exists."); + System.out.println("Nothing to delete, user does not exist."); return; } Query query = Query.keyQueryBuilder() diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java new file mode 100644 index 000000000000..f1e844c79b24 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/AddEntitiesAndRunQuery.java @@ -0,0 +1,84 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; +import com.google.gcloud.datastore.Query; +import com.google.gcloud.datastore.QueryResults; +import com.google.gcloud.datastore.StructuredQuery.PropertyFilter; + +/** + * A snippet for Google Cloud Datastore showing how to create and get entities. The snippet also + * shows how to run a query against Datastore. + */ +public class AddEntitiesAndRunQuery { + + public static void main(String... args) { + // Create datastore service object. + // By default, credentials are inferred from the runtime environment. + Datastore datastore = DatastoreOptions.defaultInstance().service(); + + // Add an entity to Datastore + KeyFactory keyFactory = datastore.newKeyFactory().kind("Person"); + Key key = keyFactory.newKey("john.doe@gmail.com"); + Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 51) + .set("favorite_food", "pizza") + .build(); + datastore.put(entity); + + // Get an entity from Datastore + Entity johnEntity = datastore.get(key); + + // Add a couple more entities to make the query results more interesting + Key janeKey = keyFactory.newKey("jane.doe@gmail.com"); + Entity janeEntity = Entity.builder(janeKey) + .set("name", "Jane Doe") + .set("age", 44) + .set("favorite_food", "pizza") + .build(); + Key joeKey = keyFactory.newKey("joe.shmoe@gmail.com"); + Entity joeEntity = Entity.builder(joeKey) + .set("name", "Joe Shmoe") + .set("age", 27) + .set("favorite_food", "sushi") + .build(); + datastore.put(janeEntity, joeEntity); + + // Run a query + Query query = Query.entityQueryBuilder() + .kind("Person") + .filter(PropertyFilter.eq("favorite_food", "pizza")) + .build(); + QueryResults results = datastore.run(query); + while (results.hasNext()) { + Entity currentEntity = results.next(); + System.out.println(currentEntity.getString("name") + ", you're invited to a pizza party!"); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java new file mode 100644 index 000000000000..3981162a2943 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/CreateEntity.java @@ -0,0 +1,48 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; + +/** + * A snippet for Google Cloud Datastore showing how to create an entity. + */ +public class CreateEntity { + + public static void main(String... args) { + Datastore datastore = DatastoreOptions.defaultInstance().service(); + KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); + Key key = keyFactory.newKey("keyName"); + Entity entity = Entity.builder(key) + .set("name", "John Doe") + .set("age", 30) + .set("access_time", DateTime.now()) + .build(); + datastore.put(entity); + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java new file mode 100644 index 000000000000..cbc97f0784dd --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/datastore/snippets/UpdateEntity.java @@ -0,0 +1,50 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.datastore.snippets; + +import com.google.gcloud.datastore.Datastore; +import com.google.gcloud.datastore.DatastoreOptions; +import com.google.gcloud.datastore.DateTime; +import com.google.gcloud.datastore.Entity; +import com.google.gcloud.datastore.Key; +import com.google.gcloud.datastore.KeyFactory; + +/** + * A snippet for Google Cloud Datastore showing how to get an entity and update it if it exists. + */ +public class UpdateEntity { + + public static void main(String... args) { + Datastore datastore = DatastoreOptions.defaultInstance().service(); + KeyFactory keyFactory = datastore.newKeyFactory().kind("keyKind"); + Key key = keyFactory.newKey("keyName"); + Entity entity = datastore.get(key); + if (entity != null) { + System.out.println("Updating access_time for " + entity.getString("name")); + entity = Entity.builder(entity) + .set("access_time", DateTime.now()) + .build(); + datastore.update(entity); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java similarity index 98% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java index 46ff82bfaf12..349c0eebe73d 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.resourcemanager; import com.google.common.base.Joiner; import com.google.gcloud.resourcemanager.Project; @@ -36,7 +36,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - {@code mvn exec:java - * -Dexec.mainClass="com.google.gcloud.examples.ResourceManagerExample" + * -Dexec.mainClass="com.google.gcloud.examples.resourcemanager.ResourceManagerExample" * -Dexec.args="[list | [create | delete | get] projectId]"}
  • * */ diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java new file mode 100644 index 000000000000..5a298107cc60 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java @@ -0,0 +1,49 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.resourcemanager.snippets; + +import com.google.gcloud.resourcemanager.Project; +import com.google.gcloud.resourcemanager.ProjectInfo; +import com.google.gcloud.resourcemanager.ResourceManager; +import com.google.gcloud.resourcemanager.ResourceManagerOptions; + +/** + * A snippet for Google Cloud Resource Manager showing how to create a project if it does not exist. + */ +public class GetOrCreateProject { + + public static void main(String... args) { + // Create Resource Manager service object. + // By default, credentials are inferred from the runtime environment. + ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); + + String projectId = "my-globally-unique-project-id"; // Change to a unique project ID. + // Get a project from the server. + Project project = resourceManager.get(projectId); + if (project == null) { + // Create a project. + project = resourceManager.create(ProjectInfo.builder(projectId).build()); + } + System.out.println("Got project " + project.projectId() + " from the server."); + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java new file mode 100644 index 000000000000..b194de0815d5 --- /dev/null +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java @@ -0,0 +1,62 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * EDITING INSTRUCTIONS + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in + * the project's READMEs and package-info.java. + */ + +package com.google.gcloud.examples.resourcemanager.snippets; + +import com.google.gcloud.resourcemanager.Project; +import com.google.gcloud.resourcemanager.ResourceManager; +import com.google.gcloud.resourcemanager.ResourceManagerOptions; + +import java.util.Iterator; + +/** + * A snippet for Google Cloud Resource Manager showing how to update a project and list all projects + * the user has permission to view. + */ +public class UpdateAndListProjects { + + public static void main(String... args) { + // Create Resource Manager service object + // By default, credentials are inferred from the runtime environment. + ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service(); + + // Get a project from the server + Project project = resourceManager.get("some-project-id"); // Use an existing project's ID + + // Update a project + if (project != null) { + Project newProject = project.toBuilder() + .addLabel("launch-status", "in-development") + .build() + .replace(); + System.out.println("Updated the labels of project " + newProject.projectId() + + " to be " + newProject.labels()); + } + + // List all the projects you have permission to view. + Iterator projectIterator = resourceManager.list().iterateAll(); + System.out.println("Projects I can view:"); + while (projectIterator.hasNext()) { + System.out.println(projectIterator.next().projectId()); + } + } +} diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java similarity index 93% rename from gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java rename to gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java index e3bee626f49c..e73cfc427129 100644 --- a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.examples; +package com.google.gcloud.examples.storage; import com.google.gcloud.AuthCredentials; import com.google.gcloud.AuthCredentials.ServiceAccountAuthCredentials; @@ -25,7 +25,6 @@ import com.google.gcloud.storage.BlobId; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.Bucket; -import com.google.gcloud.storage.BucketInfo; import com.google.gcloud.storage.CopyWriter; import com.google.gcloud.storage.Storage; import com.google.gcloud.storage.Storage.ComposeRequest; @@ -66,7 +65,7 @@ *
  • login using gcloud SDK - {@code gcloud auth login}.
  • *
  • compile using maven - {@code mvn compile}
  • *
  • run using maven - - *
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.StorageExample"
    + * 
    {@code mvn exec:java -Dexec.mainClass="com.google.gcloud.examples.storage.StorageExample"
      *  -Dexec.args="[]
      *  list [] |
      *  info [ []] |
    @@ -133,27 +132,27 @@ public void run(Storage storage, BlobId... blobIds) {
           if (blobIds.length == 1) {
             if (blobIds[0].name().isEmpty()) {
               // get Bucket
    -          Bucket bucket = Bucket.get(storage, blobIds[0].bucket());
    +          Bucket bucket = storage.get(blobIds[0].bucket());
               if (bucket == null) {
                 System.out.println("No such bucket");
                 return;
               }
    -          System.out.println("Bucket info: " + bucket.info());
    +          System.out.println("Bucket info: " + bucket);
             } else {
               // get Blob
    -          Blob blob = Blob.get(storage, blobIds[0]);
    +          Blob blob = storage.get(blobIds[0]);
               if (blob == null) {
                 System.out.println("No such object");
                 return;
               }
    -          System.out.println("Blob info: " + blob.info());
    +          System.out.println("Blob info: " + blob);
             }
           } else {
             // use batch to get multiple blobs.
    -        List blobs = Blob.get(storage, Arrays.asList(blobIds));
    +        List blobs = storage.get(blobIds);
             for (Blob blob : blobs) {
               if (blob != null) {
    -            System.out.println(blob.info());
    +            System.out.println(blob);
               }
             }
           }
    @@ -184,7 +183,7 @@ private static class DeleteAction extends BlobsAction {
         @Override
         public void run(Storage storage, BlobId... blobIds) {
           // use batch operation
    -      List deleteResults = Blob.delete(storage, blobIds);
    +      List deleteResults = storage.delete(blobIds);
           int index = 0;
           for (Boolean deleted : deleteResults) {
             if (deleted) {
    @@ -218,20 +217,20 @@ String parse(String... args) {
         public void run(Storage storage, String bucketName) {
           if (bucketName == null) {
             // list buckets
    -        Iterator bucketInfoIterator = storage.list().iterateAll();
    -        while (bucketInfoIterator.hasNext()) {
    -          System.out.println(bucketInfoIterator.next());
    +        Iterator bucketIterator = storage.list().iterateAll();
    +        while (bucketIterator.hasNext()) {
    +          System.out.println(bucketIterator.next());
             }
           } else {
             // list a bucket's blobs
    -        Bucket bucket = Bucket.get(storage, bucketName);
    +        Bucket bucket = storage.get(bucketName);
             if (bucket == null) {
               System.out.println("No such bucket");
               return;
             }
             Iterator blobIterator = bucket.list().iterateAll();
             while (blobIterator.hasNext()) {
    -          System.out.println(blobIterator.next().info());
    +          System.out.println(blobIterator.next());
             }
           }
         }
    @@ -257,8 +256,7 @@ private void run(Storage storage, Path uploadFrom, BlobInfo blobInfo) throws IOE
           if (Files.size(uploadFrom) > 1_000_000) {
             // When content is not available or large (1MB or more) it is recommended
             // to write it in chunks via the blob's channel writer.
    -        Blob blob = new Blob(storage, blobInfo);
    -        try (WriteChannel writer = blob.writer()) {
    +        try (WriteChannel writer = storage.writer(blobInfo)) {
               byte[] buffer = new byte[1024];
               try (InputStream input = Files.newInputStream(uploadFrom)) {
                 int limit;
    @@ -311,7 +309,7 @@ public void run(Storage storage, Tuple tuple) throws IOException {
         }
     
         private void run(Storage storage, BlobId blobId, Path downloadTo) throws IOException {
    -      Blob blob = Blob.get(storage, blobId);
    +      Blob blob = storage.get(blobId);
           if (blob == null) {
             System.out.println("No such object");
             return;
    @@ -320,7 +318,7 @@ private void run(Storage storage, BlobId blobId, Path downloadTo) throws IOExcep
           if (downloadTo != null) {
             writeTo = new PrintStream(new FileOutputStream(downloadTo.toFile()));
           }
    -      if (blob.info().size() < 1_000_000) {
    +      if (blob.size() < 1_000_000) {
             // Blob is small read all its content in one request
             byte[] content = blob.content();
             writeTo.write(content);
    @@ -438,13 +436,13 @@ public void run(Storage storage, Tuple> tuple)
         }
     
         private void run(Storage storage, BlobId blobId, Map metadata) {
    -      Blob blob = Blob.get(storage, blobId);
    +      Blob blob = storage.get(blobId);
           if (blob == null) {
             System.out.println("No such object");
             return;
           }
    -      Blob updateBlob = blob.update(blob.info().toBuilder().metadata(metadata).build());
    -      System.out.println("Updated " + updateBlob.info());
    +      Blob updateBlob = blob.toBuilder().metadata(metadata).build().update();
    +      System.out.println("Updated " + updateBlob);
         }
     
         @Override
    @@ -488,9 +486,8 @@ public void run(Storage storage, Tuple
           run(storage, tuple.x(), tuple.y());
         }
     
    -    private void run(Storage storage, ServiceAccountAuthCredentials cred, BlobInfo blobInfo)
    -        throws IOException {
    -      Blob blob = new Blob(storage, blobInfo);
    +    private void run(Storage storage, ServiceAccountAuthCredentials cred, BlobInfo blobInfo) {
    +      Blob blob = storage.get(blobInfo.blobId());
           System.out.println("Signed URL: "
               + blob.signUrl(1, TimeUnit.DAYS, SignUrlOption.serviceAccount(cred)));
         }
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java
    new file mode 100644
    index 000000000000..435cc90b03d8
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java
    @@ -0,0 +1,70 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.Bucket;
    +import com.google.gcloud.storage.BucketInfo;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +import java.util.Iterator;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to create a bucket and a blob in it. The snippet
    + * also shows how to get a blob's content, list buckets and list blobs.
    + */
    +public class CreateAndListBucketsAndBlobs {
    +
    +  public static void main(String... args) {
    +    // Create a service object
    +    // Credentials are inferred from the environment.
    +    Storage storage = StorageOptions.defaultInstance().service();
    +
    +    // Create a bucket
    +    String bucketName = "my_unique_bucket"; // Change this to something unique
    +    Bucket bucket = storage.create(BucketInfo.of(bucketName));
    +
    +    // Upload a blob to the newly created bucket
    +    Blob blob = bucket.create("my_blob_name", "a simple blob".getBytes(UTF_8), "text/plain");
    +
    +    // Read the blob content from the server
    +    String blobContent = new String(blob.content(), UTF_8);
    +
    +    // List all your buckets
    +    Iterator bucketIterator = storage.list().iterateAll();
    +    System.out.println("My buckets:");
    +    while (bucketIterator.hasNext()) {
    +      System.out.println(bucketIterator.next());
    +    }
    +
    +    // List the blobs in a particular bucket
    +    Iterator blobIterator = bucket.list().iterateAll();
    +    System.out.println("My blobs:");
    +    while (blobIterator.hasNext()) {
    +      System.out.println(blobIterator.next());
    +    }
    +  }
    +}
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java
    new file mode 100644
    index 000000000000..2c1304a478ab
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateBlob.java
    @@ -0,0 +1,44 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.BlobId;
    +import com.google.gcloud.storage.BlobInfo;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to create a blob.
    + */
    +public class CreateBlob {
    +
    +  public static void main(String... args) {
    +    Storage storage = StorageOptions.defaultInstance().service();
    +    BlobId blobId = BlobId.of("bucket", "blob_name");
    +    BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    +    Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    +  }
    +}
    diff --git a/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java
    new file mode 100644
    index 000000000000..13290b201787
    --- /dev/null
    +++ b/gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/UpdateBlob.java
    @@ -0,0 +1,53 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +/*
    + * EDITING INSTRUCTIONS
    + * This file is referenced in READMEs and javadoc. Any change to this file should be reflected in
    + * the project's READMEs and package-info.java.
    + */
    +
    +package com.google.gcloud.examples.storage.snippets;
    +
    +import static java.nio.charset.StandardCharsets.UTF_8;
    +
    +import com.google.gcloud.storage.Blob;
    +import com.google.gcloud.storage.BlobId;
    +import com.google.gcloud.storage.Storage;
    +import com.google.gcloud.storage.StorageOptions;
    +
    +import java.io.IOException;
    +import java.nio.ByteBuffer;
    +import java.nio.channels.WritableByteChannel;
    +
    +/**
    + * A snippet for Google Cloud Storage showing how to update the blob's content if the blob exists.
    + */
    +public class UpdateBlob {
    +
    +  public static void main(String... args) throws IOException {
    +    Storage storage = StorageOptions.defaultInstance().service();
    +    BlobId blobId = BlobId.of("bucket", "blob_name");
    +    Blob blob = storage.get(blobId);
    +    if (blob != null) {
    +      byte[] prevContent = blob.content();
    +      System.out.println(new String(prevContent, UTF_8));
    +      WritableByteChannel channel = blob.writer();
    +      channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
    +      channel.close();
    +    }
    +  }
    +}
    diff --git a/gcloud-java-resourcemanager/README.md b/gcloud-java-resourcemanager/README.md
    index d9a99e12b7a5..94037e27a709 100644
    --- a/gcloud-java-resourcemanager/README.md
    +++ b/gcloud-java-resourcemanager/README.md
    @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Resource Manager] (https://cloud.google.
     [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java)
     [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master)
     [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-resourcemanager.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-resourcemanager.svg)
    +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java)
    +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969)
     
     -  [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
     -  [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/resourcemanager/package-summary.html)
    @@ -20,21 +22,21 @@ If you are using Maven, add this to your pom.xml file
     
       com.google.gcloud
       gcloud-java-resourcemanager
    -  0.1.3
    +  0.1.5
     
     ```
     If you are using Gradle, add this to your dependencies
     ```Groovy
    -compile 'com.google.gcloud:gcloud-java-resourcemanager:0.1.3'
    +compile 'com.google.gcloud:gcloud-java-resourcemanager:0.1.5'
     ```
     If you are using SBT, add this to your dependencies
     ```Scala
    -libraryDependencies += "com.google.gcloud" % "gcloud-java-resourcemanager" % "0.1.3"
    +libraryDependencies += "com.google.gcloud" % "gcloud-java-resourcemanager" % "0.1.5"
     ```
     
     Example Application
     --------------------
    -[`ResourceManagerExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/ResourceManagerExample.java) is a simple command line interface for the Cloud Resource Manager.  Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/ResourceManagerExample.html).
    +[`ResourceManagerExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/ResourceManagerExample.java) is a simple command line interface for the Cloud Resource Manager.  Read more about using the application on the [`ResourceManagerExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/resourcemanager/ResourceManagerExample.html).
     
     Authentication
     --------------
    @@ -70,7 +72,12 @@ You will need to set up the local development environment by [installing the Goo
     You'll need to obtain the `gcloud-java-resourcemanager` library.  See the [Quickstart](#quickstart) section to add `gcloud-java-resourcemanager` as a dependency in your code.
     
     #### Creating an authorized service object
    -To make authenticated requests to Google Cloud Resource Manager, you must create a service object with Google Cloud SDK credentials. You can then make API calls by calling methods on the Resource Manager service object. The simplest way to authenticate is to use [Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials). These credentials are automatically inferred from your environment, so you only need the following code to create your service object:
    +To make authenticated requests to Google Cloud Resource Manager, you must create a service object
    +with Google Cloud SDK credentials. You can then make API calls by calling methods on the Resource
    +Manager service object. The simplest way to authenticate is to use
    +[Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
    +These credentials are automatically inferred from your environment, so you only need the following
    +code to create your service object:
     
     ```java
     import com.google.gcloud.resourcemanager.ResourceManager;
    @@ -79,46 +86,68 @@ import com.google.gcloud.resourcemanager.ResourceManagerOptions;
     ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
     ```
     
    -#### Creating a project
    -All you need to create a project is a globally unique project ID.  You can also optionally attach a non-unique name and labels to your project. Read more about naming guidelines for project IDs, names, and labels [here](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects). To create a project, add the following import at the top of your file:
    +#### Getting a specific project
    +You can load a project if you know it's project ID and have read permissions to the project.
    +To get a project, add the following import at the top of your file:
     
     ```java
     import com.google.gcloud.resourcemanager.Project;
    -import com.google.gcloud.resourcemanager.ProjectInfo;
     ```
     
    -Then add the following code to create a project (be sure to change `myProjectId` to your own unique project ID).
    +Then use the following code to get the project:
     
     ```java
    -String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    -Project myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    +String projectId = "my-globally-unique-project-id"; // Change to a unique project ID
    +Project project = resourceManager.get(projectId);
     ```
     
    -Note that the return value from `create` is a `Project` that includes additional read-only information, like creation time, project number, and lifecycle state. Read more about these fields on the [Projects page](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects). `Project`, a subclass of `ProjectInfo`, adds a layer of service-related functionality over `ProjectInfo`.
    +#### Creating a project
    +All you need to create a project is a globally unique project ID. You can also optionally attach a
    +non-unique name and labels to your project. Read more about naming guidelines for project IDs,
    +names, and labels [here](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects).
    +To create a project, add the following imports at the top of your file:
     
    -#### Getting a specific project
    -You can load a project if you know it's project ID and have read permissions to the project. For example, to get the project we just created we can do the following:
    +```java
    +import com.google.gcloud.resourcemanager.Project;
    +import com.google.gcloud.resourcemanager.ProjectInfo;
    +```
    +
    +Then add the following code to create a project (be sure to change `projectId` to your own unique
    +project ID).
     
     ```java
    -Project projectFromServer = resourceManager.get(myProjectId);
    +String projectId = "my-globally-unique-project-id"; // Change to a unique project ID
    +Project project = resourceManager.create(ProjectInfo.builder(projectId).build());
     ```
     
    +Note that the return value from `create` is a `Project` that includes additional read-only
    +information, like creation time, project number, and lifecycle state. Read more about these fields
    +on the [Projects page](https://cloud.google.com/resource-manager/reference/rest/v1beta1/projects).
    +`Project`, a subclass of `ProjectInfo`, adds a layer of service-related functionality over
    +`ProjectInfo`.
    +
     #### Editing a project
     To edit a project, create a new `ProjectInfo` object and pass it in to the `Project.replace` method.
    -
    -For example, to add a label for the newly created project to denote that it's launch status is "in development", add the following code:
    +For example, to add a label to a project to denote that it's launch status is "in development", add
    +the following code:
     
     ```java
    -Project newProject = myProject.toBuilder()
    +Project newProject = project.toBuilder()
         .addLabel("launch-status", "in-development")
         .build()
         .replace();
     ```
     
    -Note that the values of the project you pass in to `replace` overwrite the server's values for non-read-only fields, namely `projectName` and `labels`. For example, if you create a project with `projectName` "some-project-name" and subsequently call replace using a `ProjectInfo` object that didn't set the `projectName`, then the server will unset the project's name. The server ignores any attempted changes to the read-only fields `projectNumber`, `lifecycleState`, and `createTime`. The `projectId` cannot change.
    +Note that the values of the project you pass in to `replace` overwrite the server's values for
    +non-read-only fields, namely `projectName` and `labels`. For example, if you create a project with
    +`projectName` "some-project-name" and subsequently call replace using a `ProjectInfo` object that
    +didn't set the `projectName`, then the server will unset the project's name. The server ignores any
    +attempted changes to the read-only fields `projectNumber`, `lifecycleState`, and `createTime`.
    +The `projectId` cannot change.
     
     #### Listing all projects
    -Suppose that we want a list of all projects for which we have read permissions. Add the following import:
    +Suppose that we want a list of all projects for which we have read permissions. Add the following
    +import:
     
     ```java
     import java.util.Iterator;
    @@ -136,48 +165,15 @@ while (projectIterator.hasNext()) {
     
     #### Complete source code
     
    -Here we put together all the code shown above into one program.  This program assumes that you are running from your own desktop and used the Google Cloud SDK to authenticate yourself.
    -
    -```java
    -import com.google.gcloud.resourcemanager.Project;
    -import com.google.gcloud.resourcemanager.ProjectInfo;
    -import com.google.gcloud.resourcemanager.ResourceManager;
    -import com.google.gcloud.resourcemanager.ResourceManagerOptions;
    +We put together all the code shown above into two programs. Both programs assume that you are
    +running from your own desktop and used the Google Cloud SDK to authenticate yourself.
     
    -import java.util.Iterator;
    +The first program creates a project if it does not exist. Complete source code can be found at
    +[GetOrCreateProject.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/GetOrCreateProject.java).
     
    -public class GcloudJavaResourceManagerExample {
    -
    -  public static void main(String[] args) {
    -    // Create Resource Manager service object.
    -    // By default, credentials are inferred from the runtime environment.
    -    ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    -
    -    // Create a project.
    -    String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    -    Project myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    -
    -    // Get a project from the server.
    -    Project projectFromServer = resourceManager.get(myProjectId);
    -    System.out.println("Got project " + projectFromServer.projectId() + " from the server.");
    -
    -    // Update a project
    -    Project newProject = myProject.toBuilder()
    -        .addLabel("launch-status", "in-development")
    -        .build()
    -        .replace();
    -    System.out.println("Updated the labels of project " + newProject.projectId()
    -        + " to be " + newProject.labels());
    -
    -    // List all the projects you have permission to view.
    -    Iterator projectIterator = resourceManager.list().iterateAll();
    -    System.out.println("Projects I can view:");
    -    while (projectIterator.hasNext()) {
    -      System.out.println(projectIterator.next().projectId());
    -    }
    -  }
    -}
    -```
    +The second program updates a project if it exists and lists all projects the user has permission to
    +view. Complete source code can be found at
    +[UpdateAndListProjects.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/resourcemanager/snippets/UpdateAndListProjects.java).
     
     Java Versions
     -------------
    diff --git a/gcloud-java-resourcemanager/pom.xml b/gcloud-java-resourcemanager/pom.xml
    index 8fc6dd723eef..c10691d3b07d 100644
    --- a/gcloud-java-resourcemanager/pom.xml
    +++ b/gcloud-java-resourcemanager/pom.xml
    @@ -1,7 +1,6 @@
     
     
       4.0.0
    -  com.google.gcloud
       gcloud-java-resourcemanager
       jar
       GCloud Java resource manager
    @@ -11,7 +10,7 @@
       
         com.google.gcloud
         gcloud-java-pom
    -    0.1.4-SNAPSHOT
    +    0.1.6-SNAPSHOT
       
       
         gcloud-java-resourcemanager
    @@ -25,7 +24,7 @@
         
           com.google.apis
           google-api-services-cloudresourcemanager
    -      v1beta1-rev6-1.19.0
    +      v1beta1-rev10-1.21.0
           compile
           
             
    @@ -43,7 +42,7 @@
         
           org.easymock
           easymock
    -      3.3
    +      3.4
           test
         
       
    diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java
    new file mode 100644
    index 000000000000..0d7118dcbbd7
    --- /dev/null
    +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Policy.java
    @@ -0,0 +1,164 @@
    +/*
    + * Copyright 2016 Google Inc. All Rights Reserved.
    + *
    + * Licensed under the Apache License, Version 2.0 (the "License");
    + * you may not use this file except in compliance with the License.
    + * You may obtain a copy of the License at
    + *
    + *       http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package com.google.gcloud.resourcemanager;
    +
    +import com.google.common.annotations.VisibleForTesting;
    +import com.google.common.base.CaseFormat;
    +import com.google.common.base.Function;
    +import com.google.common.collect.ImmutableSet;
    +import com.google.common.collect.Lists;
    +import com.google.gcloud.IamPolicy;
    +import com.google.gcloud.Identity;
    +
    +import java.util.ArrayList;
    +import java.util.HashMap;
    +import java.util.LinkedList;
    +import java.util.List;
    +import java.util.Map;
    +import java.util.Set;
    +
    +/**
    + * An Identity and Access Management (IAM) policy for a project. IAM policies are used to specify
    + * access settings for Cloud Platform resources. A policy is a map of bindings. A binding assigns
    + * a set of identities to a role, where the identities can be user accounts, Google groups, Google
    + * domains, and service accounts. A role is a named list of permissions defined by IAM. Policies set
    + * at the project level control access both to the project and to resources associated with the
    + * project.
    + *
    + * @see Policy
    + */
    +public class Policy extends IamPolicy {
    +
    +  private static final long serialVersionUID = -5573557282693961850L;
    +
    +  /**
    +   * Represents legacy roles in an IAM Policy.
    +   */
    +  public enum Role {
    +
    +    /**
    +     * Permissions for read-only actions that preserve state.
    +     */
    +    VIEWER("roles/viewer"),
    +
    +    /**
    +     * All viewer permissions and permissions for actions that modify state.
    +     */
    +    EDITOR("roles/editor"),
    +
    +    /**
    +     * All editor permissions and permissions for the following actions:
    +     * 
      + *
    • Manage access control for a resource. + *
    • Set up billing (for a project). + *
    + */ + OWNER("roles/owner"); + + private String strValue; + + private Role(String strValue) { + this.strValue = strValue; + } + + String strValue() { + return strValue; + } + + static Role fromStr(String roleStr) { + return Role.valueOf(CaseFormat.LOWER_CAMEL.to( + CaseFormat.UPPER_UNDERSCORE, roleStr.substring("roles/".length()))); + } + } + + /** + * Builder for an IAM Policy. + */ + public static class Builder extends IamPolicy.Builder { + + private Builder() {} + + @VisibleForTesting + Builder(Map> bindings, String etag, Integer version) { + bindings(bindings).etag(etag).version(version); + } + + @Override + public Policy build() { + return new Policy(this); + } + } + + private Policy(Builder builder) { + super(builder); + } + + public static Builder builder() { + return new Builder(); + } + + @Override + public Builder toBuilder() { + return new Builder(bindings(), etag(), version()); + } + + com.google.api.services.cloudresourcemanager.model.Policy toPb() { + com.google.api.services.cloudresourcemanager.model.Policy policyPb = + new com.google.api.services.cloudresourcemanager.model.Policy(); + List bindingPbList = + new LinkedList<>(); + for (Map.Entry> binding : bindings().entrySet()) { + com.google.api.services.cloudresourcemanager.model.Binding bindingPb = + new com.google.api.services.cloudresourcemanager.model.Binding(); + bindingPb.setRole(binding.getKey().strValue()); + bindingPb.setMembers( + Lists.transform( + new ArrayList<>(binding.getValue()), + new Function() { + @Override + public String apply(Identity identity) { + return identity.strValue(); + } + })); + bindingPbList.add(bindingPb); + } + policyPb.setBindings(bindingPbList); + policyPb.setEtag(etag()); + policyPb.setVersion(version()); + return policyPb; + } + + static Policy fromPb( + com.google.api.services.cloudresourcemanager.model.Policy policyPb) { + Map> bindings = new HashMap<>(); + for (com.google.api.services.cloudresourcemanager.model.Binding bindingPb : + policyPb.getBindings()) { + bindings.put( + Role.fromStr(bindingPb.getRole()), + ImmutableSet.copyOf( + Lists.transform( + bindingPb.getMembers(), + new Function() { + @Override + public Identity apply(String identityPb) { + return Identity.valueOf(identityPb); + } + }))); + } + return new Policy.Builder(bindings, policyPb.getEtag(), policyPb.getVersion()).build(); + } +} diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java index f12a7ea50676..4d12a31274c0 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/Project.java @@ -38,14 +38,12 @@ public class Project extends ProjectInfo { private final ResourceManagerOptions options; private transient ResourceManager resourceManager; + /** + * Builder for {@code Project}. + */ public static class Builder extends ProjectInfo.Builder { private final ResourceManager resourceManager; - private ProjectInfo.BuilderImpl infoBuilder; - - Builder(ResourceManager resourceManager) { - this.resourceManager = resourceManager; - this.infoBuilder = new ProjectInfo.BuilderImpl(); - } + private final ProjectInfo.BuilderImpl infoBuilder; Builder(Project project) { this.resourceManager = project.resourceManager; @@ -124,16 +122,6 @@ public Project build() { this.options = resourceManager.options(); } - /** - * Constructs a Project object that contains project information got from the server. - * - * @return Project object containing the project's metadata or {@code null} if not found - * @throws ResourceManagerException upon failure - */ - public static Project get(ResourceManager resourceManager, String projectId) { - return resourceManager.get(projectId); - } - /** * Returns the {@link ResourceManager} service object associated with this Project. */ @@ -142,14 +130,14 @@ public ResourceManager resourceManager() { } /** - * Fetches the current project's latest information. Returns {@code null} if the job does not + * Fetches the project's latest information. Returns {@code null} if the project does not * exist. * * @return Project containing the project's updated metadata or {@code null} if not found * @throws ResourceManagerException upon failure */ public Project reload() { - return Project.get(resourceManager, projectId()); + return resourceManager.get(projectId()); } /** @@ -210,10 +198,6 @@ public Project replace() { return resourceManager.replace(this); } - static Builder builder(ResourceManager resourceManager, String projectId) { - return new Builder(resourceManager).projectId(projectId); - } - @Override public Builder toBuilder() { return new Builder(this); diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java index 7553a207cd29..260e8a8e2f26 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ProjectInfo.java @@ -115,7 +115,10 @@ static ResourceId fromPb( } } - public static abstract class Builder { + /** + * Builder for {@code ProjectInfo}. + */ + public abstract static class Builder { /** * Set the user-assigned name of the project. @@ -184,7 +187,9 @@ static class BuilderImpl extends Builder { private Long createTimeMillis; private ResourceId parent; - BuilderImpl() {} + BuilderImpl(String projectId) { + this.projectId = projectId; + } BuilderImpl(ProjectInfo info) { this.name = info.name; @@ -331,7 +336,7 @@ public Long createTimeMillis() { @Override public boolean equals(Object obj) { - return obj.getClass().equals(ProjectInfo.class) + return obj != null && obj.getClass().equals(ProjectInfo.class) && Objects.equals(toPb(), ((ProjectInfo) obj).toPb()); } @@ -341,7 +346,7 @@ public int hashCode() { } public static Builder builder(String id) { - return new BuilderImpl().projectId(id); + return new BuilderImpl(id); } public Builder toBuilder() { diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java index af772dce6b60..f9ddf6872414 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManager.java @@ -147,8 +147,6 @@ public static ProjectListOption pageToken(String pageToken) { * *

    The server can return fewer projects than requested. When there are more results than the * page size, the server will return a page token that can be used to fetch other results. - * Note: pagination is not yet supported; the server currently ignores this field and returns - * all results. */ public static ProjectListOption pageSize(int pageSize) { return new ProjectListOption(ResourceManagerRpc.Option.PAGE_SIZE, pageSize); @@ -164,13 +162,13 @@ public static ProjectListOption pageSize(int pageSize) { */ public static ProjectListOption fields(ProjectField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("projects(").append(ProjectField.selector(fields)).append(")"); + builder.append("projects(").append(ProjectField.selector(fields)).append("),nextPageToken"); return new ProjectListOption(ResourceManagerRpc.Option.FIELDS, builder.toString()); } } /** - * Create a new project. + * Creates a new project. * *

    Initially, the project resource is owned by its creator exclusively. The creator can later * grant permission to others to read or update the project. Several APIs are activated @@ -228,8 +226,7 @@ public static ProjectListOption fields(ProjectField... fields) { * *

    This method returns projects in an unspecified order. New projects do not necessarily appear * at the end of the list. Use {@link ProjectListOption} to filter this list, set page size, and - * set page tokens. Note that pagination is currently not implemented by the Cloud Resource - * Manager API. + * set page tokens. * * @see Cloud diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java index e087caab5966..4176b4e610ba 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/ResourceManagerImpl.java @@ -140,7 +140,8 @@ Iterable> call() { public Project apply( com.google.api.services.cloudresourcemanager.model.Project projectPb) { return new Project( - serviceOptions.service(), new ProjectInfo.BuilderImpl(ProjectInfo.fromPb(projectPb))); + serviceOptions.service(), + new ProjectInfo.BuilderImpl(ProjectInfo.fromPb(projectPb))); } }); return new PageImpl<>( diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java index 22a81499eb7a..d1794447e9fb 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/package-info.java @@ -17,21 +17,40 @@ /** * A client to Google Cloud Resource Manager. * - *

    Here's a simple usage example for using gcloud-java-resourcemanager: + *

    Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * creates a project if it does not exist. For the complete source code see + * + * GetOrCreateProject.java. *

     {@code
      * ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    - * String myProjectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    - * Project myProject = resourceManager.create(ProjectInfo.builder(myProjectId).build());
    - * Project newProject = myProject.toBuilder()
    - *     .addLabel("launch-status", "in-development")
    - *     .build()
    - *     .replace();
    + * String projectId = "my-globally-unique-project-id"; // Change to a unique project ID.
    + * Project project = resourceManager.get(projectId);
    + * if (project == null) {
    + *   project = resourceManager.create(ProjectInfo.builder(projectId).build());
    + * }
    + * System.out.println("Got project " + project.projectId() + " from the server.");
    + * }
    + *

    + * This second example shows how to update a project if it exists and list all projects the user has + * permission to view. For the complete source code see + * + * UpdateAndListProjects.java. + *

     {@code
    + * ResourceManager resourceManager = ResourceManagerOptions.defaultInstance().service();
    + * Project project = resourceManager.get("some-project-id"); // Use an existing project's ID
    + * if (project != null) {
    + *   Project newProject = project.toBuilder()
    + *       .addLabel("launch-status", "in-development")
    + *       .build()
    + *       .replace();
    + *   System.out.println("Updated the labels of project " + newProject.projectId()
    + *       + " to be " + newProject.labels());
    + * }
      * Iterator projectIterator = resourceManager.list().iterateAll();
      * System.out.println("Projects I can view:");
      * while (projectIterator.hasNext()) {
      *   System.out.println(projectIterator.next().projectId());
      * }}
    - * *

    Remember that you must authenticate using the Google Cloud SDK. See more about * providing * credentials here. diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java index 25c763276b3b..cda2dd1e00ea 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/resourcemanager/testing/LocalResourceManagerHelper.java @@ -12,6 +12,7 @@ import com.google.common.collect.ImmutableMap; import com.google.common.collect.ImmutableSet; import com.google.common.io.ByteStreams; +import com.google.gcloud.AuthCredentials; import com.google.gcloud.resourcemanager.ResourceManagerOptions; import com.sun.net.httpserver.Headers; @@ -34,9 +35,11 @@ import java.util.Map; import java.util.Random; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentSkipListMap; import java.util.logging.Level; import java.util.logging.Logger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import java.util.zip.GZIPInputStream; /** @@ -56,6 +59,9 @@ public class LocalResourceManagerHelper { private static final URI BASE_CONTEXT; private static final Set SUPPORTED_COMPRESSION_ENCODINGS = ImmutableSet.of("gzip", "x-gzip"); + private static final Pattern LIST_FIELDS_PATTERN = + Pattern.compile("(.*?)projects\\((.*?)\\)(.*?)"); + private static final String[] NO_FIELDS = {}; static { try { @@ -71,7 +77,7 @@ public class LocalResourceManagerHelper { ImmutableSet.of('-', '\'', '"', ' ', '!'); private final HttpServer server; - private final ConcurrentHashMap projects = new ConcurrentHashMap<>(); + private final ConcurrentSkipListMap projects = new ConcurrentSkipListMap<>(); private final int port; private static class Response { @@ -228,7 +234,7 @@ private static String[] parseFields(String query) { return null; } - private static Map parseListOptions(String query) { + private static Map parseListOptions(String query) throws IOException { Map options = new HashMap<>(); if (query != null) { String[] args = query.split("&"); @@ -236,19 +242,28 @@ private static Map parseListOptions(String query) { String[] argEntry = arg.split("="); switch (argEntry[0]) { case "fields": - // List fields are in the form "projects(field1, field2, ...)" - options.put( - "fields", - argEntry[1].substring("projects(".length(), argEntry[1].length() - 1).split(",")); + // List fields are in the form "projects(field1, field2, ...),nextPageToken" + Matcher matcher = LIST_FIELDS_PATTERN.matcher(argEntry[1]); + if (matcher.matches()) { + options.put("projectFields", matcher.group(2).split(",")); + options.put("listFields", (matcher.group(1) + matcher.group(3)).split(",")); + } else { + options.put("projectFields", NO_FIELDS); + options.put("listFields", argEntry[1].split(",")); + } break; case "filter": options.put("filter", argEntry[1].split(" ")); break; case "pageToken": - // support pageToken when Cloud Resource Manager supports this (#421) + options.put("pageToken", argEntry[1]); break; case "pageSize": - // support pageSize when Cloud Resource Manager supports this (#421) + int pageSize = Integer.valueOf(argEntry[1]); + if (pageSize < 1) { + throw new IOException("Page size must be greater than 0."); + } + options.put("pageSize", pageSize); break; } } @@ -353,28 +368,55 @@ Response get(String projectId, String[] fields) { } Response list(Map options) { - // Use pageSize and pageToken options when Cloud Resource Manager does so (#421) List projectsSerialized = new ArrayList<>(); String[] filters = (String[]) options.get("filter"); if (filters != null && !isValidFilter(filters)) { return Error.INVALID_ARGUMENT.response("Could not parse the filter."); } - String[] fields = (String[]) options.get("fields"); - for (Project p : projects.values()) { + String[] projectFields = (String[]) options.get("projectFields"); + int count = 0; + String pageToken = (String) options.get("pageToken"); + Integer pageSize = (Integer) options.get("pageSize"); + String nextPageToken = null; + Map projectsToScan = projects; + if (pageToken != null) { + projectsToScan = projects.tailMap(pageToken); + } + for (Project p : projectsToScan.values()) { + if (pageSize != null && count >= pageSize) { + nextPageToken = p.getProjectId(); + break; + } boolean includeProject = includeProject(p, filters); if (includeProject) { + count++; try { - projectsSerialized.add(jsonFactory.toString(extractFields(p, fields))); + projectsSerialized.add(jsonFactory.toString(extractFields(p, projectFields))); } catch (IOException e) { return Error.INTERNAL_ERROR.response( "Error when serializing project " + p.getProjectId()); } } } + String[] listFields = (String[]) options.get("listFields"); StringBuilder responseBody = new StringBuilder(); - responseBody.append("{\"projects\": ["); - Joiner.on(",").appendTo(responseBody, projectsSerialized); - responseBody.append("]}"); + responseBody.append('{'); + // If fields parameter is set but no project field is selected we must return no projects. + if (!(projectFields != null && projectFields.length == 0)) { + responseBody.append("\"projects\": ["); + Joiner.on(",").appendTo(responseBody, projectsSerialized); + responseBody.append(']'); + } + if (nextPageToken != null && (listFields == null + || ImmutableSet.copyOf(listFields).contains("nextPageToken"))) { + if (responseBody.length() > 1) { + responseBody.append(','); + } + responseBody.append("\"nextPageToken\": \""); + responseBody.append(nextPageToken); + responseBody.append('"'); + } + responseBody.append('}'); return new Response(HTTP_OK, responseBody.toString()); } @@ -509,17 +551,21 @@ private LocalResourceManagerHelper() { } /** - * Creates a LocalResourceManagerHelper object that listens to requests on the local machine. + * Creates a {@code LocalResourceManagerHelper} object that listens to requests on the local + * machine. */ public static LocalResourceManagerHelper create() { return new LocalResourceManagerHelper(); } /** - * Returns a ResourceManagerOptions instance that sets the host to use the mock server. + * Returns a {@link ResourceManagerOptions} instance that sets the host to use the mock server. */ public ResourceManagerOptions options() { - return ResourceManagerOptions.builder().host("http://localhost:" + port).build(); + return ResourceManagerOptions.builder() + .host("http://localhost:" + port) + .authCredentials(AuthCredentials.noAuth()) + .build(); } /** diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/DefaultResourceManagerRpc.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/DefaultResourceManagerRpc.java index 61c622fa0c33..d30cd2df3627 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/DefaultResourceManagerRpc.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/DefaultResourceManagerRpc.java @@ -38,7 +38,7 @@ private static ResourceManagerException translate(IOException exception) { } @Override - public Project create(Project project) throws ResourceManagerException { + public Project create(Project project) { try { return resourceManager.projects().create(project).execute(); } catch (IOException ex) { @@ -47,7 +47,7 @@ public Project create(Project project) throws ResourceManagerException { } @Override - public void delete(String projectId) throws ResourceManagerException { + public void delete(String projectId) { try { resourceManager.projects().delete(projectId).execute(); } catch (IOException ex) { @@ -56,7 +56,7 @@ public void delete(String projectId) throws ResourceManagerException { } @Override - public Project get(String projectId, Map options) throws ResourceManagerException { + public Project get(String projectId, Map options) { try { return resourceManager.projects() .get(projectId) @@ -74,8 +74,7 @@ public Project get(String projectId, Map options) throws ResourceMana } @Override - public Tuple> list(Map options) - throws ResourceManagerException { + public Tuple> list(Map options) { try { ListProjectsResponse response = resourceManager.projects() .list() @@ -92,7 +91,7 @@ public Tuple> list(Map options) } @Override - public void undelete(String projectId) throws ResourceManagerException { + public void undelete(String projectId) { try { resourceManager.projects().undelete(projectId).execute(); } catch (IOException ex) { @@ -101,7 +100,7 @@ public void undelete(String projectId) throws ResourceManagerException { } @Override - public Project replace(Project project) throws ResourceManagerException { + public Project replace(Project project) { try { return resourceManager.projects().update(project.getProjectId(), project).execute(); } catch (IOException ex) { diff --git a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/ResourceManagerRpc.java b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/ResourceManagerRpc.java index 52dfc2d2368e..e1d72a6a373e 100644 --- a/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/ResourceManagerRpc.java +++ b/gcloud-java-resourcemanager/src/main/java/com/google/gcloud/spi/ResourceManagerRpc.java @@ -75,17 +75,51 @@ public Y y() { } } - Project create(Project project) throws ResourceManagerException; - - void delete(String projectId) throws ResourceManagerException; - - Project get(String projectId, Map options) throws ResourceManagerException; - - Tuple> list(Map options) throws ResourceManagerException; - - void undelete(String projectId) throws ResourceManagerException; - - Project replace(Project project) throws ResourceManagerException; + /** + * Creates a new project. + * + * @throws ResourceManagerException upon failure + */ + Project create(Project project); + + /** + * Marks the project identified by the specified project ID for deletion. + * + * @throws ResourceManagerException upon failure + */ + void delete(String projectId); + + /** + * Retrieves the project identified by the specified project ID. Returns {@code null} if the + * project is not found or if the user doesn't have read permissions for the project. + * + * @throws ResourceManagerException upon failure + */ + Project get(String projectId, Map options); + + /** + * Lists the projects visible to the current user. + * + * @throws ResourceManagerException upon failure + */ + Tuple> list(Map options); + + /** + * Restores the project identified by the specified project ID. Undelete will only succeed if the + * project has a lifecycle state of {@code DELETE_REQUESTED} state. The caller must have modify + * permissions for this project. + * + * @throws ResourceManagerException upon failure + */ + void undelete(String projectId); + + /** + * Replaces the attributes of the project. The caller must have modify permissions for this + * project. + * + * @throws ResourceManagerException upon failure + */ + Project replace(Project project); // TODO(ajaykannan): implement "Organization" functionality when available (issue #319) } diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java index 7eb0156d4e56..a3418fff98ab 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/LocalResourceManagerHelperTest.java @@ -19,6 +19,7 @@ import org.junit.Test; import java.util.HashMap; +import java.util.Iterator; import java.util.Map; public class LocalResourceManagerHelperTest { @@ -278,7 +279,7 @@ public void testGetWithOptions() { public void testList() { Tuple> projects = rpc.list(EMPTY_RPC_OPTIONS); - assertNull(projects.x()); // change this when #421 is resolved + assertNull(projects.x()); assertFalse(projects.y().iterator().hasNext()); rpc.create(COMPLETE_PROJECT); RESOURCE_MANAGER_HELPER.changeLifecycleState( @@ -297,11 +298,43 @@ public void testList() { } @Test - public void testListFieldOptions() { + public void testInvalidListPaging() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, -1); + try { + rpc.list(rpcOptions); + } catch (ResourceManagerException e) { + assertEquals("Page size must be greater than 0.", e.getMessage()); + } + } + + @Test + public void testListPaging() { Map rpcOptions = new HashMap<>(); - rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "projects(projectId,name,labels)"); - rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, "somePageToken"); rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + compareReadWriteFields(COMPLETE_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + iterator = projects.y().iterator(); + compareReadWriteFields(PARTIAL_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + assertNull(projects.x()); + } + + @Test + public void testListFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, + "projects(projectId,name,labels),nextPageToken"); rpc.create(PROJECT_WITH_PARENT); Tuple> projects = rpc.list(rpcOptions); @@ -317,6 +350,81 @@ public void testListFieldOptions() { assertNull(returnedProject.getCreateTime()); } + @Test + public void testListPageTokenFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "nextPageToken,projects(projectId,name)"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + com.google.api.services.cloudresourcemanager.model.Project returnedProject = iterator.next(); + assertEquals(COMPLETE_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(COMPLETE_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertFalse(iterator.hasNext()); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + iterator = projects.y().iterator(); + returnedProject = iterator.next(); + assertEquals(PARTIAL_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(PARTIAL_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertNull(projects.x()); + } + + @Test + public void testListNoPageTokenFieldOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "projects(projectId,name)"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNull(projects.x()); + Iterator iterator = + projects.y().iterator(); + com.google.api.services.cloudresourcemanager.model.Project returnedProject = iterator.next(); + assertEquals(COMPLETE_PROJECT.getProjectId(), returnedProject.getProjectId()); + assertEquals(COMPLETE_PROJECT.getName(), returnedProject.getName()); + assertNull(returnedProject.getLabels()); + assertNull(returnedProject.getParent()); + assertNull(returnedProject.getProjectNumber()); + assertNull(returnedProject.getLifecycleState()); + assertNull(returnedProject.getCreateTime()); + assertFalse(iterator.hasNext()); + } + + @Test + public void testListPageTokenNoFieldsOptions() { + Map rpcOptions = new HashMap<>(); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_SIZE, 1); + rpcOptions.put(ResourceManagerRpc.Option.FIELDS, "nextPageToken"); + rpc.create(PARTIAL_PROJECT); + rpc.create(COMPLETE_PROJECT); + Tuple> projects = + rpc.list(rpcOptions); + assertNotNull(projects.x()); + assertNull(projects.y()); + rpcOptions.put(ResourceManagerRpc.Option.PAGE_TOKEN, projects.x()); + projects = rpc.list(rpcOptions); + assertNull(projects.x()); + assertNull(projects.y()); + } + @Test public void testListFilterOptions() { Map rpcFilterOptions = new HashMap<>(); diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java new file mode 100644 index 000000000000..05d1b85bdbed --- /dev/null +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/PolicyTest.java @@ -0,0 +1,53 @@ +/* + * Copyright 2016 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.resourcemanager; + +import static org.junit.Assert.assertEquals; + +import com.google.common.collect.ImmutableSet; +import com.google.gcloud.Identity; + +import org.junit.Test; + +public class PolicyTest { + + private static final Identity ALL_USERS = Identity.allUsers(); + private static final Identity ALL_AUTH_USERS = Identity.allAuthenticatedUsers(); + private static final Identity USER = Identity.user("abc@gmail.com"); + private static final Identity SERVICE_ACCOUNT = + Identity.serviceAccount("service-account@gmail.com"); + private static final Identity GROUP = Identity.group("group@gmail.com"); + private static final Identity DOMAIN = Identity.domain("google.com"); + private static final Policy SIMPLE_POLICY = Policy.builder() + .addBinding(Policy.Role.VIEWER, ImmutableSet.of(USER, SERVICE_ACCOUNT, ALL_USERS)) + .addBinding(Policy.Role.EDITOR, ImmutableSet.of(ALL_AUTH_USERS, GROUP, DOMAIN)) + .build(); + private static final Policy FULL_POLICY = + new Policy.Builder(SIMPLE_POLICY.bindings(), "etag", 1).build(); + + @Test + public void testIamPolicyToBuilder() { + assertEquals(FULL_POLICY, FULL_POLICY.toBuilder().build()); + assertEquals(SIMPLE_POLICY, SIMPLE_POLICY.toBuilder().build()); + } + + @Test + public void testPolicyToAndFromPb() { + assertEquals(FULL_POLICY, Policy.fromPb(FULL_POLICY.toPb())); + assertEquals(SIMPLE_POLICY, Policy.fromPb(SIMPLE_POLICY.toPb())); + } +} diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java index a741963913c6..4e239acc45ef 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ProjectTest.java @@ -24,11 +24,11 @@ import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; import com.google.common.collect.ImmutableMap; import org.junit.After; +import org.junit.Before; import org.junit.Test; import java.util.Map; @@ -54,6 +54,11 @@ public class ProjectTest { private Project expectedProject; private Project project; + @Before + public void setUp() { + resourceManager = createStrictMock(ResourceManager.class); + } + @After public void tearDown() throws Exception { verify(resourceManager); @@ -62,7 +67,6 @@ public void tearDown() throws Exception { private void initializeExpectedProject(int optionsCalls) { expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); replay(serviceMockReturnsOptions); - resourceManager = createStrictMock(ResourceManager.class); expectedProject = new Project(serviceMockReturnsOptions, new ProjectInfo.BuilderImpl(PROJECT_INFO)); } @@ -71,31 +75,33 @@ private void initializeProject() { project = new Project(resourceManager, new ProjectInfo.BuilderImpl(PROJECT_INFO)); } + @Test + public void testToBuilder() { + initializeExpectedProject(4); + replay(resourceManager); + compareProjects(expectedProject, expectedProject.toBuilder().build()); + } + @Test public void testBuilder() { - initializeExpectedProject(2); + initializeExpectedProject(4); + expect(resourceManager.options()).andReturn(mockOptions).times(4); replay(resourceManager); - Project builtProject = Project.builder(serviceMockReturnsOptions, PROJECT_ID) - .name(NAME) + Project.Builder builder = + new Project.Builder(new Project(resourceManager, new ProjectInfo.BuilderImpl(PROJECT_ID))); + Project project = builder.name(NAME) .labels(LABELS) .projectNumber(PROJECT_NUMBER) .createTimeMillis(CREATE_TIME_MILLIS) .state(STATE) .build(); - assertEquals(PROJECT_ID, builtProject.projectId()); - assertEquals(NAME, builtProject.name()); - assertEquals(LABELS, builtProject.labels()); - assertEquals(PROJECT_NUMBER, builtProject.projectNumber()); - assertEquals(CREATE_TIME_MILLIS, builtProject.createTimeMillis()); - assertEquals(STATE, builtProject.state()); - assertSame(serviceMockReturnsOptions, builtProject.resourceManager()); - } - - @Test - public void testToBuilder() { - initializeExpectedProject(4); - replay(resourceManager); - compareProjects(expectedProject, expectedProject.toBuilder().build()); + assertEquals(PROJECT_ID, project.projectId()); + assertEquals(NAME, project.name()); + assertEquals(LABELS, project.labels()); + assertEquals(PROJECT_NUMBER, project.projectNumber()); + assertEquals(CREATE_TIME_MILLIS, project.createTimeMillis()); + assertEquals(STATE, project.state()); + assertEquals(resourceManager.options(), project.resourceManager().options()); } @Test @@ -103,7 +109,7 @@ public void testGet() { initializeExpectedProject(1); expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(expectedProject); replay(resourceManager); - Project loadedProject = Project.get(resourceManager, PROJECT_INFO.projectId()); + Project loadedProject = resourceManager.get(PROJECT_INFO.projectId()); assertEquals(expectedProject, loadedProject); } @@ -126,7 +132,7 @@ public void testLoadNull() { initializeExpectedProject(1); expect(resourceManager.get(PROJECT_INFO.projectId())).andReturn(null); replay(resourceManager); - assertNull(Project.get(resourceManager, PROJECT_INFO.projectId())); + assertNull(resourceManager.get(PROJECT_INFO.projectId())); } @Test diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java index 37c54718fb4a..1bc233311a4d 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/ResourceManagerImplTest.java @@ -42,6 +42,7 @@ import org.junit.Test; import org.junit.rules.ExpectedException; +import java.util.Iterator; import java.util.Map; public class ResourceManagerImplTest { @@ -166,7 +167,7 @@ public void testGetWithOptions() { @Test public void testList() { Page projects = RESOURCE_MANAGER.list(); - assertFalse(projects.values().iterator().hasNext()); // TODO: change this when #421 is resolved + assertFalse(projects.values().iterator().hasNext()); RESOURCE_MANAGER.create(PARTIAL_PROJECT); RESOURCE_MANAGER.create(COMPLETE_PROJECT); for (Project p : RESOURCE_MANAGER.list().values()) { @@ -181,6 +182,22 @@ public void testList() { } } + @Test + public void testListPaging() { + RESOURCE_MANAGER.create(PARTIAL_PROJECT); + RESOURCE_MANAGER.create(COMPLETE_PROJECT); + Page page = RESOURCE_MANAGER.list(ProjectListOption.pageSize(1)); + assertNotNull(page.nextPageCursor()); + Iterator iterator = page.values().iterator(); + compareReadWriteFields(COMPLETE_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + page = page.nextPage(); + iterator = page.values().iterator(); + compareReadWriteFields(PARTIAL_PROJECT, iterator.next()); + assertFalse(iterator.hasNext()); + assertNull(page.nextPageCursor()); + } + @Test public void testListFieldOptions() { RESOURCE_MANAGER.create(COMPLETE_PROJECT); @@ -196,6 +213,38 @@ public void testListFieldOptions() { assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); } + @Test + public void testListPagingWithFieldOptions() { + RESOURCE_MANAGER.create(PARTIAL_PROJECT); + RESOURCE_MANAGER.create(COMPLETE_PROJECT); + Page projects = RESOURCE_MANAGER.list(LIST_FIELDS, ProjectListOption.pageSize(1)); + assertNotNull(projects.nextPageCursor()); + Iterator iterator = projects.values().iterator(); + Project returnedProject = iterator.next(); + assertEquals(COMPLETE_PROJECT.projectId(), returnedProject.projectId()); + assertEquals(COMPLETE_PROJECT.name(), returnedProject.name()); + assertEquals(COMPLETE_PROJECT.labels(), returnedProject.labels()); + assertNull(returnedProject.parent()); + assertNull(returnedProject.projectNumber()); + assertNull(returnedProject.state()); + assertNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); + assertFalse(iterator.hasNext()); + projects = projects.nextPage(); + iterator = projects.values().iterator(); + returnedProject = iterator.next(); + assertEquals(PARTIAL_PROJECT.projectId(), returnedProject.projectId()); + assertEquals(PARTIAL_PROJECT.name(), returnedProject.name()); + assertEquals(PARTIAL_PROJECT.labels(), returnedProject.labels()); + assertNull(returnedProject.parent()); + assertNull(returnedProject.projectNumber()); + assertNull(returnedProject.state()); + assertNull(returnedProject.createTimeMillis()); + assertSame(RESOURCE_MANAGER, returnedProject.resourceManager()); + assertFalse(iterator.hasNext()); + assertNull(projects.nextPageCursor()); + } + @Test public void testListFilterOptions() { ProjectInfo matchingProject = ProjectInfo.builder("matching-project") diff --git a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java index 497de880254a..35b72ae1713f 100644 --- a/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java +++ b/gcloud-java-resourcemanager/src/test/java/com/google/gcloud/resourcemanager/SerializationTest.java @@ -20,6 +20,8 @@ import static org.junit.Assert.assertNotSame; import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.gcloud.Identity; import com.google.gcloud.PageImpl; import com.google.gcloud.RetryParams; @@ -53,6 +55,9 @@ public class SerializationTest { ResourceManager.ProjectGetOption.fields(ResourceManager.ProjectField.NAME); private static final ResourceManager.ProjectListOption PROJECT_LIST_OPTION = ResourceManager.ProjectListOption.filter("name:*"); + private static final Policy POLICY = Policy.builder() + .addBinding(Policy.Role.VIEWER, ImmutableSet.of(Identity.user("abc@gmail.com"))) + .build(); @Test public void testServiceOptions() throws Exception { @@ -70,7 +75,7 @@ public void testServiceOptions() throws Exception { @Test public void testModelAndRequests() throws Exception { Serializable[] objects = {PARTIAL_PROJECT_INFO, FULL_PROJECT_INFO, PROJECT, PAGE_RESULT, - PROJECT_GET_OPTION, PROJECT_LIST_OPTION}; + PROJECT_GET_OPTION, PROJECT_LIST_OPTION, POLICY}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-storage/README.md b/gcloud-java-storage/README.md index 7260ab5fe5c5..0ee05b31c10c 100644 --- a/gcloud-java-storage/README.md +++ b/gcloud-java-storage/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Storage] (https://cloud.google.com/stora [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-storage.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java-storage.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/storage/package-summary.html) @@ -20,22 +22,22 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java-storage - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java-storage:0.1.3' +compile 'com.google.gcloud:gcloud-java-storage:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java-storage" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java-storage" % "0.1.5" ``` Example Application ------------------- -[`StorageExample`](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java) is a simple command line interface that provides some of Cloud Storage's functionality. Read more about using the application on the [`gcloud-java-examples` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/StorageExample.html). +[`StorageExample`](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/StorageExample.java) is a simple command line interface that provides some of Cloud Storage's functionality. Read more about using the application on the [`StorageExample` docs page](http://googlecloudplatform.github.io/gcloud-java/apidocs/?com/google/gcloud/examples/storage/StorageExample.html). Authentication -------------- @@ -77,15 +79,15 @@ Storage storage = StorageOptions.defaultInstance().service(); For other authentication options, see the [Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page. #### Storing data -Stored objects are called "blobs" in `gcloud-java` and are organized into containers called "buckets". In this code snippet, we will create a new bucket and upload a blob to that bucket. +Stored objects are called "blobs" in `gcloud-java` and are organized into containers called "buckets". `Blob`, a subclass of `BlobInfo`, adds a layer of service-related functionality over `BlobInfo`. Similarly, `Bucket` adds a layer of service-related functionality over `BucketInfo`. In this code snippet, we will create a new bucket and upload a blob to that bucket. Add the following imports at the top of your file: ```java import static java.nio.charset.StandardCharsets.UTF_8; -import com.google.gcloud.storage.BlobId; -import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.Bucket; import com.google.gcloud.storage.BucketInfo; ``` @@ -96,13 +98,12 @@ Then add the following code to create a bucket and upload a simple blob. ```java // Create a bucket String bucketName = "my_unique_bucket"; // Change this to something unique -BucketInfo bucketInfo = storage.create(BucketInfo.of(bucketName)); +Bucket bucket = storage.create(BucketInfo.of(bucketName)); // Upload a blob to the newly created bucket BlobId blobId = BlobId.of(bucketName, "my_blob_name"); -BlobInfo blobInfo = storage.create( - BlobInfo.builder(blobId).contentType("text/plain").build(), - "a simple blob".getBytes(UTF_8)); +Blob blob = storage.create( + "my_blob_name", "a simple blob".getBytes(UTF_8), "text/plain"); ``` At this point, you will be able to see your newly created bucket and blob on the Google Developers Console. @@ -111,7 +112,7 @@ At this point, you will be able to see your newly created bucket and blob on the Now that we have content uploaded to the server, we can see how to read data from the server. Add the following line to your program to get back the blob we uploaded. ```java -String blobContent = new String(storage.readAllBytes(blobId), UTF_8); +String blobContent = new String(blob.content(), UTF_8); ``` #### Listing buckets and contents of buckets @@ -125,14 +126,14 @@ Then add the following code to list all your buckets and all the blobs inside yo ```java // List all your buckets -Iterator bucketInfoIterator = storage.list().iterateAll(); +Iterator bucketIterator = storage.list().iterateAll(); System.out.println("My buckets:"); -while (bucketInfoIterator.hasNext()) { - System.out.println(bucketInfoIterator.next()); +while (bucketIterator.hasNext()) { + System.out.println(bucketIterator.next()); } // List the blobs in a particular bucket -Iterator blobIterator = storage.list(bucketName).iterateAll(); +Iterator blobIterator = bucket.list().iterateAll(); System.out.println("My blobs:"); while (blobIterator.hasNext()) { System.out.println(blobIterator.next()); @@ -141,55 +142,12 @@ while (blobIterator.hasNext()) { #### Complete source code -Here we put together all the code shown above into one program. This program assumes that you are running on Compute Engine or from your own desktop. To run this example on App Engine, simply move the code from the main method to your application's servlet class and change the print statements to display on your webpage. - -```java -import static java.nio.charset.StandardCharsets.UTF_8; - -import com.google.gcloud.storage.BlobId; -import com.google.gcloud.storage.BlobInfo; -import com.google.gcloud.storage.BucketInfo; -import com.google.gcloud.storage.Storage; -import com.google.gcloud.storage.StorageOptions; - -import java.util.Iterator; - -public class GcloudStorageExample { - - public static void main(String[] args) { - // Create a service object - // Credentials are inferred from the environment. - Storage storage = StorageOptions.defaultInstance().service(); - - // Create a bucket - String bucketName = "my_unique_bucket"; // Change this to something unique - BucketInfo bucketInfo = storage.create(BucketInfo.of(bucketName)); - - // Upload a blob to the newly created bucket - BlobId blobId = BlobId.of(bucketName, "my_blob_name"); - BlobInfo blobInfo = storage.create( - BlobInfo.builder(blobId).contentType("text/plain").build(), - "a simple blob".getBytes(UTF_8)); - - // Retrieve a blob from the server - String blobContent = new String(storage.readAllBytes(blobId), UTF_8); - - // List all your buckets - Iterator bucketInfoIterator = storage.list().iterateAll(); - System.out.println("My buckets:"); - while (bucketInfoIterator.hasNext()) { - System.out.println(bucketInfoIterator.next()); - } - - // List the blobs in a particular bucket - Iterator blobIterator = storage.list(bucketName).iterateAll(); - System.out.println("My blobs:"); - while (blobIterator.hasNext()) { - System.out.println(blobIterator.next()); - } - } -} -``` +In +[CreateAndListBucketsAndBlobs.java](../gcloud-java-examples/src/main/java/com/google/gcloud/examples/storage/snippets/CreateAndListBucketsAndBlobs.java) +we put together all the code shown above into one program. The program assumes that you are +running on Compute Engine or from your own desktop. To run the example on App Engine, simply move +the code from the main method to your application's servlet class and change the print statements to +display on your webpage. Troubleshooting --------------- diff --git a/gcloud-java-storage/pom.xml b/gcloud-java-storage/pom.xml index 4e9d368a12bb..d5f0f6d98660 100644 --- a/gcloud-java-storage/pom.xml +++ b/gcloud-java-storage/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java-storage jar GCloud Java storage @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT gcloud-java-storage @@ -25,13 +24,13 @@ com.google.apis google-api-services-storage - v1-rev33-1.20.0 + v1-rev62-1.21.0 compile - - com.google.guava - guava-jdk5 - + + com.google.guava + guava-jdk5 + com.google.api-client google-api-client @@ -47,7 +46,7 @@ org.easymock easymock - 3.3 + 3.4 test diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java b/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java index dc84a1de5559..cac47b4bd248 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/spi/DefaultStorageRpc.java @@ -31,6 +31,7 @@ import static com.google.gcloud.spi.StorageRpc.Option.PREFIX; import static com.google.gcloud.spi.StorageRpc.Option.VERSIONS; import static java.net.HttpURLConnection.HTTP_NOT_FOUND; +import static javax.servlet.http.HttpServletResponse.SC_REQUESTED_RANGE_NOT_SATISFIABLE; import com.google.api.client.googleapis.batch.json.JsonBatchCallback; import com.google.api.client.googleapis.json.GoogleJsonError; @@ -99,7 +100,7 @@ private static StorageException translate(GoogleJsonError exception) { } @Override - public Bucket create(Bucket bucket, Map options) throws StorageException { + public Bucket create(Bucket bucket, Map options) { try { return storage.buckets() .insert(this.options.projectId(), bucket) @@ -114,7 +115,7 @@ public Bucket create(Bucket bucket, Map options) throws StorageExcept @Override public StorageObject create(StorageObject storageObject, final InputStream content, - Map options) throws StorageException { + Map options) { try { Storage.Objects.Insert insert = storage.objects() .insert(storageObject.getBucket(), storageObject, @@ -296,7 +297,7 @@ private Storage.Objects.Delete deleteRequest(StorageObject blob, Map @Override public StorageObject compose(Iterable sources, StorageObject target, - Map targetOptions) throws StorageException { + Map targetOptions) { ComposeRequest request = new ComposeRequest(); if (target.getContentType() == null) { target.setContentType("application/octet-stream"); @@ -327,8 +328,7 @@ public StorageObject compose(Iterable sources, StorageObject targ } @Override - public byte[] load(StorageObject from, Map options) - throws StorageException { + public byte[] load(StorageObject from, Map options) { try { Storage.Objects.Get getRequest = storage.objects() .get(from.getBucket(), from.getName()) @@ -347,7 +347,7 @@ public byte[] load(StorageObject from, Map options) } @Override - public BatchResponse batch(BatchRequest request) throws StorageException { + public BatchResponse batch(BatchRequest request) { List>>> partitionedToDelete = Lists.partition(request.toDelete, MAX_BATCH_DELETES); Iterator>>> iterator = partitionedToDelete.iterator(); @@ -437,7 +437,7 @@ public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) { @Override public Tuple read(StorageObject from, Map options, long position, - int bytes) throws StorageException { + int bytes) { try { Get req = storage.objects() .get(from.getBucket(), from.getName()) @@ -454,20 +454,32 @@ public Tuple read(StorageObject from, Map options, lo String etag = req.getLastResponseHeaders().getETag(); return Tuple.of(etag, output.toByteArray()); } catch (IOException ex) { - throw translate(ex); + StorageException serviceException = translate(ex); + if (serviceException.code() == SC_REQUESTED_RANGE_NOT_SATISFIABLE) { + return Tuple.of(null, new byte[0]); + } + throw serviceException; } } @Override public void write(String uploadId, byte[] toWrite, int toWriteOffset, long destOffset, int length, - boolean last) throws StorageException { + boolean last) { try { + if (length == 0 && !last) { + return; + } GenericUrl url = new GenericUrl(uploadId); HttpRequest httpRequest = storage.getRequestFactory().buildPutRequest(url, new ByteArrayContent(null, toWrite, toWriteOffset, length)); long limit = destOffset + length; StringBuilder range = new StringBuilder("bytes "); - range.append(destOffset).append('-').append(limit - 1).append('/'); + if (length == 0) { + range.append('*'); + } else { + range.append(destOffset).append('-').append(limit - 1); + } + range.append('/'); if (last) { range.append(limit); } else { @@ -501,8 +513,7 @@ public void write(String uploadId, byte[] toWrite, int toWriteOffset, long destO } @Override - public String open(StorageObject object, Map options) - throws StorageException { + public String open(StorageObject object, Map options) { try { Insert req = storage.objects().insert(object.getBucket(), object); GenericUrl url = req.buildHttpRequest().getUrl(); @@ -538,16 +549,16 @@ public String open(StorageObject object, Map options) } @Override - public RewriteResponse openRewrite(RewriteRequest rewriteRequest) throws StorageException { + public RewriteResponse openRewrite(RewriteRequest rewriteRequest) { return rewrite(rewriteRequest, null); } @Override - public RewriteResponse continueRewrite(RewriteResponse previousResponse) throws StorageException { + public RewriteResponse continueRewrite(RewriteResponse previousResponse) { return rewrite(previousResponse.rewriteRequest, previousResponse.rewriteToken); } - private RewriteResponse rewrite(RewriteRequest req, String token) throws StorageException { + private RewriteResponse rewrite(RewriteRequest req, String token) { try { Long maxBytesRewrittenPerCall = req.megabytesRewrittenPerCall != null ? req.megabytesRewrittenPerCall * MEGABYTE : null; diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/spi/StorageRpc.java b/gcloud-java-storage/src/main/java/com/google/gcloud/spi/StorageRpc.java index e15a27114810..c76fc0f2d3b8 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/spi/StorageRpc.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/spi/StorageRpc.java @@ -217,57 +217,134 @@ public int hashCode() { } } - Bucket create(Bucket bucket, Map options) throws StorageException; + /** + * Creates a new bucket. + * + * @throws StorageException upon failure + */ + Bucket create(Bucket bucket, Map options); - StorageObject create(StorageObject object, InputStream content, Map options) - throws StorageException; + /** + * Creates a new storage object. + * + * @throws StorageException upon failure + */ + StorageObject create(StorageObject object, InputStream content, Map options); - Tuple> list(Map options) throws StorageException; + /** + * Lists the project's buckets. + * + * @throws StorageException upon failure + */ + Tuple> list(Map options); - Tuple> list(String bucket, Map options) - throws StorageException; + /** + * Lists the bucket's blobs. + * + * @throws StorageException upon failure + */ + Tuple> list(String bucket, Map options); /** * Returns the requested bucket or {@code null} if not found. * * @throws StorageException upon failure */ - Bucket get(Bucket bucket, Map options) throws StorageException; + Bucket get(Bucket bucket, Map options); /** * Returns the requested storage object or {@code null} if not found. * * @throws StorageException upon failure */ - StorageObject get(StorageObject object, Map options) - throws StorageException; + StorageObject get(StorageObject object, Map options); - Bucket patch(Bucket bucket, Map options) throws StorageException; + /** + * Updates bucket information. + * + * @throws StorageException upon failure + */ + Bucket patch(Bucket bucket, Map options); - StorageObject patch(StorageObject storageObject, Map options) - throws StorageException; + /** + * Updates the storage object's information. Original metadata are merged with metadata in the + * provided {@code storageObject}. + * + * @throws StorageException upon failure + */ + StorageObject patch(StorageObject storageObject, Map options); - boolean delete(Bucket bucket, Map options) throws StorageException; + /** + * Deletes the requested bucket. + * + * @return {@code true} if the bucket was deleted, {@code false} if it was not found + * @throws StorageException upon failure + */ + boolean delete(Bucket bucket, Map options); - boolean delete(StorageObject object, Map options) throws StorageException; + /** + * Deletes the requested storage object. + * + * @return {@code true} if the storage object was deleted, {@code false} if it was not found + * @throws StorageException upon failure + */ + boolean delete(StorageObject object, Map options); - BatchResponse batch(BatchRequest request) throws StorageException; + /** + * Sends a batch request. + * + * @throws StorageException upon failure + */ + BatchResponse batch(BatchRequest request); + /** + * Sends a compose request. + * + * @throws StorageException upon failure + */ StorageObject compose(Iterable sources, StorageObject target, - Map targetOptions) throws StorageException; + Map targetOptions); - byte[] load(StorageObject storageObject, Map options) - throws StorageException; + /** + * Reads all the bytes from a storage object. + * + * @throws StorageException upon failure + */ + byte[] load(StorageObject storageObject, Map options); - Tuple read(StorageObject from, Map options, long position, int bytes) - throws StorageException; + /** + * Reads the given amount of bytes from a storage object at the given position. + * + * @throws StorageException upon failure + */ + Tuple read(StorageObject from, Map options, long position, int bytes); - String open(StorageObject object, Map options) throws StorageException; + /** + * Opens a resumable upload channel for a given storage object. + * + * @throws StorageException upon failure + */ + String open(StorageObject object, Map options); + /** + * Writes the provided bytes to a storage object at the provided location. + * + * @throws StorageException upon failure + */ void write(String uploadId, byte[] toWrite, int toWriteOffset, long destOffset, int length, - boolean last) throws StorageException; + boolean last); - RewriteResponse openRewrite(RewriteRequest rewriteRequest) throws StorageException; + /** + * Sends a rewrite request to open a rewrite channel. + * + * @throws StorageException upon failure + */ + RewriteResponse openRewrite(RewriteRequest rewriteRequest); - RewriteResponse continueRewrite(RewriteResponse previousResponse) throws StorageException; + /** + * Continues rewriting on an already open rewrite channel. + * + * @throws StorageException + */ + RewriteResponse continueRewrite(RewriteResponse previousResponse); } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java index 98e7ce09cef0..fe5f6f5743c8 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BatchResponse.java @@ -31,8 +31,8 @@ public final class BatchResponse implements Serializable { private static final long serialVersionUID = 1057416839397037706L; private final List> deleteResult; - private final List> updateResult; - private final List> getResult; + private final List> updateResult; + private final List> getResult; public static class Result implements Serializable { @@ -113,8 +113,8 @@ static Result empty() { } } - BatchResponse(List> deleteResult, List> updateResult, - List> getResult) { + BatchResponse(List> deleteResult, List> updateResult, + List> getResult) { this.deleteResult = ImmutableList.copyOf(deleteResult); this.updateResult = ImmutableList.copyOf(updateResult); this.getResult = ImmutableList.copyOf(getResult); @@ -146,14 +146,14 @@ public List> deletes() { /** * Returns the results for the update operations using the request order. */ - public List> updates() { + public List> updates() { return updateResult; } /** * Returns the results for the get operations using the request order. */ - public List> gets() { + public List> gets() { return getResult; } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java index fe65f6ee010b..aea424ca4063 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Blob.java @@ -16,26 +16,27 @@ package com.google.gcloud.storage; -import static com.google.common.base.Preconditions.checkArgument; import static com.google.common.base.Preconditions.checkNotNull; import static com.google.gcloud.storage.Blob.BlobSourceOption.toGetOptions; import static com.google.gcloud.storage.Blob.BlobSourceOption.toSourceOptions; +import com.google.api.services.storage.model.StorageObject; import com.google.common.base.Function; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.Lists; import com.google.gcloud.ReadChannel; import com.google.gcloud.WriteChannel; import com.google.gcloud.spi.StorageRpc; +import com.google.gcloud.spi.StorageRpc.Tuple; import com.google.gcloud.storage.Storage.BlobTargetOption; import com.google.gcloud.storage.Storage.BlobWriteOption; import com.google.gcloud.storage.Storage.CopyRequest; import com.google.gcloud.storage.Storage.SignUrlOption; +import java.io.IOException; +import java.io.ObjectInputStream; import java.net.URL; import java.util.Arrays; -import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.Objects; import java.util.concurrent.TimeUnit; @@ -44,13 +45,24 @@ * *

    Objects of this class are immutable. Operations that modify the blob like {@link #update} and * {@link #copyTo} return a new object. To get a {@code Blob} object with the most recent - * information use {@link #reload}. + * information use {@link #reload}. {@code Blob} adds a layer of service-related functionality over + * {@link BlobInfo}. *

    */ -public final class Blob { +public final class Blob extends BlobInfo { - private final Storage storage; - private final BlobInfo info; + private static final long serialVersionUID = -6806832496717441434L; + + private final StorageOptions options; + private transient Storage storage; + + static final Function, Blob> BLOB_FROM_PB_FUNCTION = + new Function, Blob>() { + @Override + public Blob apply(Tuple pb) { + return Blob.fromPb(pb.x(), pb.y()); + } + }; /** * Class for specifying blob source options when {@code Blob} methods are used. @@ -146,60 +158,148 @@ static Storage.BlobGetOption[] toGetOptions(BlobInfo blobInfo, BlobSourceOption. } /** - * Constructs a {@code Blob} object for the provided {@code BlobInfo}. The storage service is used - * to issue requests. - * - * @param storage the storage service used for issuing requests - * @param info blob's info + * Builder for {@code Blob}. */ - public Blob(Storage storage, BlobInfo info) { - this.storage = checkNotNull(storage); - this.info = checkNotNull(info); - } + public static class Builder extends BlobInfo.Builder { - /** - * Creates a {@code Blob} object for the provided bucket and blob names. Performs an RPC call to - * get the latest blob information. Returns {@code null} if the blob does not exist. - * - * @param storage the storage service used for issuing requests - * @param bucket bucket's name - * @param options blob get options - * @param blob blob's name - * @return the {@code Blob} object or {@code null} if not found - * @throws StorageException upon failure - */ - public static Blob get(Storage storage, String bucket, String blob, - Storage.BlobGetOption... options) { - return get(storage, BlobId.of(bucket, blob), options); - } + private final Storage storage; + private final BlobInfo.BuilderImpl infoBuilder; - /** - * Creates a {@code Blob} object for the provided {@code blobId}. Performs an RPC call to get the - * latest blob information. Returns {@code null} if the blob does not exist. - * - * @param storage the storage service used for issuing requests - * @param blobId blob's identifier - * @param options blob get options - * @return the {@code Blob} object or {@code null} if not found - * @throws StorageException upon failure - */ - public static Blob get(Storage storage, BlobId blobId, Storage.BlobGetOption... options) { - BlobInfo info = storage.get(blobId, options); - return info != null ? new Blob(storage, info) : null; - } + Builder(Blob blob) { + this.storage = blob.storage(); + this.infoBuilder = new BlobInfo.BuilderImpl(blob); + } - /** - * Returns the blob's information. - */ - public BlobInfo info() { - return info; + @Override + public Builder blobId(BlobId blobId) { + infoBuilder.blobId(blobId); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + public Builder contentType(String contentType) { + infoBuilder.contentType(contentType); + return this; + } + + @Override + public Builder contentDisposition(String contentDisposition) { + infoBuilder.contentDisposition(contentDisposition); + return this; + } + + @Override + public Builder contentLanguage(String contentLanguage) { + infoBuilder.contentLanguage(contentLanguage); + return this; + } + + @Override + public Builder contentEncoding(String contentEncoding) { + infoBuilder.contentEncoding(contentEncoding); + return this; + } + + @Override + Builder componentCount(Integer componentCount) { + infoBuilder.componentCount(componentCount); + return this; + } + + @Override + public Builder cacheControl(String cacheControl) { + infoBuilder.cacheControl(cacheControl); + return this; + } + + @Override + public Builder acl(List acl) { + infoBuilder.acl(acl); + return this; + } + + @Override + Builder owner(Acl.Entity owner) { + infoBuilder.owner(owner); + return this; + } + + @Override + Builder size(Long size) { + infoBuilder.size(size); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder md5(String md5) { + infoBuilder.md5(md5); + return this; + } + + @Override + public Builder crc32c(String crc32c) { + infoBuilder.crc32c(crc32c); + return this; + } + + @Override + Builder mediaLink(String mediaLink) { + infoBuilder.mediaLink(mediaLink); + return this; + } + + @Override + public Builder metadata(Map metadata) { + infoBuilder.metadata(metadata); + return this; + } + + @Override + Builder metageneration(Long metageneration) { + infoBuilder.metageneration(metageneration); + return this; + } + + @Override + Builder deleteTime(Long deleteTime) { + infoBuilder.deleteTime(deleteTime); + return this; + } + + @Override + Builder updateTime(Long updateTime) { + infoBuilder.updateTime(updateTime); + return this; + } + + @Override + public Blob build() { + return new Blob(storage, infoBuilder); + } } - /** - * Returns the blob's id. - */ - public BlobId id() { - return info.blobId(); + Blob(Storage storage, BlobInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.storage = checkNotNull(storage); + this.options = storage.options(); } /** @@ -211,9 +311,9 @@ public BlobId id() { */ public boolean exists(BlobSourceOption... options) { int length = options.length; - Storage.BlobGetOption[] getOptions = Arrays.copyOf(toGetOptions(info, options), length + 1); + Storage.BlobGetOption[] getOptions = Arrays.copyOf(toGetOptions(this, options), length + 1); getOptions[length] = Storage.BlobGetOption.fields(); - return storage.get(info.blobId(), getOptions) != null; + return storage.get(blobId(), getOptions) != null; } /** @@ -223,7 +323,7 @@ public boolean exists(BlobSourceOption... options) { * @throws StorageException upon failure */ public byte[] content(Storage.BlobSourceOption... options) { - return storage.readAllBytes(info.blobId(), options); + return storage.readAllBytes(blobId(), options); } /** @@ -234,7 +334,7 @@ public byte[] content(Storage.BlobSourceOption... options) { * @throws StorageException upon failure */ public Blob reload(BlobSourceOption... options) { - return Blob.get(storage, info.blobId(), toGetOptions(info, options)); + return storage.get(blobId(), toGetOptions(this, options)); } /** @@ -243,27 +343,26 @@ public Blob reload(BlobSourceOption... options) { * {@link #delete} operations. A new {@code Blob} object is returned. By default no checks are * made on the metadata generation of the current blob. If you want to update the information only * if the current blob metadata are at their latest version use the {@code metagenerationMatch} - * option: {@code blob.update(newInfo, BlobTargetOption.metagenerationMatch())}. + * option: {@code newBlob.update(BlobTargetOption.metagenerationMatch())}. * - *

    Original metadata are merged with metadata in the provided {@code blobInfo}. To replace - * metadata instead you first have to unset them. Unsetting metadata can be done by setting the - * provided {@code blobInfo}'s metadata to {@code null}. + *

    Original metadata are merged with metadata in the provided in this {@code blob}. To replace + * metadata instead you first have to unset them. Unsetting metadata can be done by setting this + * {@code blob}'s metadata to {@code null}. *

    * *

    Example usage of replacing blob's metadata: - *

        {@code blob.update(blob.info().toBuilder().metadata(null).build());}
    -   *    {@code blob.update(blob.info().toBuilder().metadata(newMetadata).build());}
    +   * 
     {@code
    +   * blob.toBuilder().metadata(null).build().update();
    +   * blob.toBuilder().metadata(newMetadata).build().update();
    +   * }
        * 
    * - * @param blobInfo new blob's information. Bucket and blob names must match the current ones * @param options update options * @return a {@code Blob} object with updated information * @throws StorageException upon failure */ - public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { - checkArgument(Objects.equals(blobInfo.bucket(), info.bucket()), "Bucket name must match"); - checkArgument(Objects.equals(blobInfo.name(), info.name()), "Blob name must match"); - return new Blob(storage, storage.update(blobInfo, options)); + public Blob update(BlobTargetOption... options) { + return storage.update(this, options); } /** @@ -274,7 +373,7 @@ public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { * @throws StorageException upon failure */ public boolean delete(BlobSourceOption... options) { - return storage.delete(info.blobId(), toSourceOptions(info, options)); + return storage.delete(blobId(), toSourceOptions(this, options)); } /** @@ -288,8 +387,11 @@ public boolean delete(BlobSourceOption... options) { * @throws StorageException upon failure */ public CopyWriter copyTo(BlobId targetBlob, BlobSourceOption... options) { - CopyRequest copyRequest = CopyRequest.builder().source(info.bucket(), info.name()) - .sourceOptions(toSourceOptions(info, options)).target(targetBlob).build(); + CopyRequest copyRequest = CopyRequest.builder() + .source(bucket(), name()) + .sourceOptions(toSourceOptions(this, options)) + .target(targetBlob) + .build(); return storage.copy(copyRequest); } @@ -304,7 +406,7 @@ public CopyWriter copyTo(BlobId targetBlob, BlobSourceOption... options) { * @throws StorageException upon failure */ public CopyWriter copyTo(String targetBucket, BlobSourceOption... options) { - return copyTo(targetBucket, info.name(), options); + return copyTo(targetBucket, name(), options); } /** @@ -329,7 +431,7 @@ public CopyWriter copyTo(String targetBucket, String targetBlob, BlobSourceOptio * @throws StorageException upon failure */ public ReadChannel reader(BlobSourceOption... options) { - return storage.reader(info.blobId(), toSourceOptions(info, options)); + return storage.reader(blobId(), toSourceOptions(this, options)); } /** @@ -341,7 +443,7 @@ public ReadChannel reader(BlobSourceOption... options) { * @throws StorageException upon failure */ public WriteChannel writer(BlobWriteOption... options) { - return storage.writer(info, options); + return storage.writer(this, options); } /** @@ -358,7 +460,7 @@ public WriteChannel writer(BlobWriteOption... options) { * @see Signed-URLs */ public URL signUrl(long duration, TimeUnit unit, SignUrlOption... options) { - return storage.signUrl(info, duration, unit, options); + return storage.signUrl(this, duration, unit, options); } /** @@ -368,97 +470,29 @@ public Storage storage() { return storage; } - /** - * Gets the requested blobs. A batch request is used to fetch blobs. - * - * @param storage the storage service used to issue the request - * @param first the first blob to get - * @param second the second blob to get - * @param other other blobs to get - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List get(Storage storage, BlobId first, BlobId second, BlobId... other) { - checkNotNull(storage); - checkNotNull(first); - checkNotNull(second); - checkNotNull(other); - ImmutableList blobs = ImmutableList.builder() - .add(first) - .add(second) - .addAll(Arrays.asList(other)) - .build(); - return get(storage, blobs); + @Override + public Builder toBuilder() { + return new Builder(this); } - /** - * Gets the requested blobs. A batch request is used to fetch blobs. - * - * @param storage the storage service used to issue the request - * @param blobs list of blobs to get - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List get(final Storage storage, List blobs) { - checkNotNull(storage); - checkNotNull(blobs); - BlobId[] blobArray = blobs.toArray(new BlobId[blobs.size()]); - return Collections.unmodifiableList(Lists.transform(storage.get(blobArray), - new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return blobInfo != null ? new Blob(storage, blobInfo) : null; - } - })); + @Override + public boolean equals(Object obj) { + return obj instanceof Blob && Objects.equals(toPb(), ((Blob) obj).toPb()) + && Objects.equals(options, ((Blob) obj).options); } - /** - * Updates the requested blobs. A batch request is used to update blobs. Original metadata are - * merged with metadata in the provided {@code BlobInfo} objects. To replace metadata instead - * you first have to unset them. Unsetting metadata can be done by setting the provided - * {@code BlobInfo} objects metadata to {@code null}. See - * {@link #update(com.google.gcloud.storage.BlobInfo, - * com.google.gcloud.storage.Storage.BlobTargetOption...) } for a code example. - * - * @param storage the storage service used to issue the request - * @param infos the blobs to update - * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it has - * been denied the corresponding item in the list is {@code null} - * @throws StorageException upon failure - */ - public static List update(final Storage storage, BlobInfo... infos) { - checkNotNull(storage); - checkNotNull(infos); - if (infos.length == 0) { - return Collections.emptyList(); - } - return Collections.unmodifiableList(Lists.transform(storage.update(infos), - new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return blobInfo != null ? new Blob(storage, blobInfo) : null; - } - })); + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); } - /** - * Deletes the requested blobs. A batch request is used to delete blobs. - * - * @param storage the storage service used to issue the request - * @param blobs the blobs to delete - * @return an immutable list of booleans. If a blob has been deleted the corresponding item in the - * list is {@code true}. If a blob was not found, deletion failed or access to the resource - * was denied the corresponding item is {@code false} - * @throws StorageException upon failure - */ - public static List delete(Storage storage, BlobId... blobs) { - checkNotNull(storage); - checkNotNull(blobs); - if (blobs.length == 0) { - return Collections.emptyList(); - } - return storage.delete(blobs); + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.storage = options.service(); + } + + static Blob fromPb(Storage storage, StorageObject storageObject) { + BlobInfo info = BlobInfo.fromPb(storageObject); + return new Blob(storage, new BlobInfo.BuilderImpl(info)); } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java index b27d00d68a16..54fabe87d766 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobInfo.java @@ -47,22 +47,16 @@ * @see Concepts and * Terminology */ -public final class BlobInfo implements Serializable { +public class BlobInfo implements Serializable { - static final Function FROM_PB_FUNCTION = - new Function() { - @Override - public BlobInfo apply(StorageObject pb) { - return BlobInfo.fromPb(pb); - } - }; - static final Function TO_PB_FUNCTION = + static final Function INFO_TO_PB_FUNCTION = new Function() { @Override public StorageObject apply(BlobInfo blobInfo) { return blobInfo.toPb(); } }; + private static final long serialVersionUID = 2228487739943277159L; private final BlobId blobId; private final String id; @@ -96,7 +90,110 @@ public Set> entrySet() { } } - public static final class Builder { + /** + * Builder for {@code BlobInfo}. + */ + public abstract static class Builder { + + /** + * Sets the blob identity. + */ + public abstract Builder blobId(BlobId blobId); + + abstract Builder id(String id); + + /** + * Sets the blob's data content type. + * + * @see Content-Type + */ + public abstract Builder contentType(String contentType); + + /** + * Sets the blob's data content disposition. + * + * @see Content-Disposition + */ + public abstract Builder contentDisposition(String contentDisposition); + + /** + * Sets the blob's data content language. + * + * @see Content-Language + */ + public abstract Builder contentLanguage(String contentLanguage); + + /** + * Sets the blob's data content encoding. + * + * @see Content-Encoding + */ + public abstract Builder contentEncoding(String contentEncoding); + + abstract Builder componentCount(Integer componentCount); + + /** + * Sets the blob's data cache control. + * + * @see Cache-Control + */ + public abstract Builder cacheControl(String cacheControl); + + /** + * Sets the blob's access control configuration. + * + * @see + * About Access Control Lists + */ + public abstract Builder acl(List acl); + + abstract Builder owner(Acl.Entity owner); + + abstract Builder size(Long size); + + abstract Builder etag(String etag); + + abstract Builder selfLink(String selfLink); + + /** + * Sets the MD5 hash of blob's data. MD5 value must be encoded in base64. + * + * @see + * Hashes and ETags: Best Practices + */ + public abstract Builder md5(String md5); + + /** + * Sets the CRC32C checksum of blob's data as described in + * RFC 4960, Appendix B; encoded in + * base64 in big-endian order. + * + * @see + * Hashes and ETags: Best Practices + */ + public abstract Builder crc32c(String crc32c); + + abstract Builder mediaLink(String mediaLink); + + /** + * Sets the blob's user provided metadata. + */ + public abstract Builder metadata(Map metadata); + + abstract Builder metageneration(Long metageneration); + + abstract Builder deleteTime(Long deleteTime); + + abstract Builder updateTime(Long updateTime); + + /** + * Creates a {@code BlobInfo} object. + */ + public abstract BlobInfo build(); + } + + static final class BuilderImpl extends Builder { private BlobId blobId; private String id; @@ -119,9 +216,11 @@ public static final class Builder { private Long deleteTime; private Long updateTime; - private Builder() {} + BuilderImpl(BlobId blobId) { + this.blobId = blobId; + } - private Builder(BlobInfo blobInfo) { + BuilderImpl(BlobInfo blobInfo) { blobId = blobInfo.blobId; id = blobInfo.id; cacheControl = blobInfo.cacheControl; @@ -144,168 +243,135 @@ private Builder(BlobInfo blobInfo) { updateTime = blobInfo.updateTime; } - /** - * Sets the blob identity. - */ + @Override public Builder blobId(BlobId blobId) { this.blobId = checkNotNull(blobId); return this; } + @Override Builder id(String id) { this.id = id; return this; } - /** - * Sets the blob's data content type. - * - * @see Content-Type - */ + @Override public Builder contentType(String contentType) { this.contentType = firstNonNull(contentType, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content disposition. - * - * @see Content-Disposition - */ + @Override public Builder contentDisposition(String contentDisposition) { this.contentDisposition = firstNonNull(contentDisposition, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content language. - * - * @see Content-Language - */ + @Override public Builder contentLanguage(String contentLanguage) { this.contentLanguage = firstNonNull(contentLanguage, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's data content encoding. - * - * @see Content-Encoding - */ + @Override public Builder contentEncoding(String contentEncoding) { this.contentEncoding = firstNonNull(contentEncoding, Data.nullOf(String.class)); return this; } + @Override Builder componentCount(Integer componentCount) { this.componentCount = componentCount; return this; } - /** - * Sets the blob's data cache control. - * - * @see Cache-Control - */ + @Override public Builder cacheControl(String cacheControl) { this.cacheControl = firstNonNull(cacheControl, Data.nullOf(String.class)); return this; } - /** - * Sets the blob's access control configuration. - * - * @see - * About Access Control Lists - */ + @Override public Builder acl(List acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } + @Override Builder owner(Acl.Entity owner) { this.owner = owner; return this; } + @Override Builder size(Long size) { this.size = size; return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Sets the MD5 hash of blob's data. MD5 value must be encoded in base64. - * - * @see - * Hashes and ETags: Best Practices - */ + @Override public Builder md5(String md5) { this.md5 = firstNonNull(md5, Data.nullOf(String.class)); return this; } - /** - * Sets the CRC32C checksum of blob's data as described in - * RFC 4960, Appendix B; encoded in - * base64 in big-endian order. - * - * @see - * Hashes and ETags: Best Practices - */ + @Override public Builder crc32c(String crc32c) { this.crc32c = firstNonNull(crc32c, Data.nullOf(String.class)); return this; } + @Override Builder mediaLink(String mediaLink) { this.mediaLink = mediaLink; return this; } - /** - * Sets the blob's user provided metadata. - */ + @Override public Builder metadata(Map metadata) { this.metadata = metadata != null ? new HashMap<>(metadata) : Data.>nullOf(ImmutableEmptyMap.class); return this; } + @Override Builder metageneration(Long metageneration) { this.metageneration = metageneration; return this; } + @Override Builder deleteTime(Long deleteTime) { this.deleteTime = deleteTime; return this; } + @Override Builder updateTime(Long updateTime) { this.updateTime = updateTime; return this; } - /** - * Creates a {@code BlobInfo} object. - */ + @Override public BlobInfo build() { checkNotNull(blobId); return new BlobInfo(this); } } - private BlobInfo(Builder builder) { + BlobInfo(BuilderImpl builder) { blobId = builder.blobId; id = builder.id; cacheControl = builder.cacheControl; @@ -526,7 +592,7 @@ public Long updateTime() { * Returns a builder for the current blob. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -548,7 +614,8 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof BlobInfo && Objects.equals(toPb(), ((BlobInfo) obj).toPb()); + return obj != null && obj.getClass().equals(BlobInfo.class) + && Objects.equals(toPb(), ((BlobInfo) obj).toPb()); } StorageObject toPb() { @@ -609,7 +676,7 @@ public static Builder builder(BucketInfo bucketInfo, String name) { * Returns a {@code BlobInfo} builder where blob identity is set using the provided values. */ public static Builder builder(String bucket, String name) { - return new Builder().blobId(BlobId.of(bucket, name)); + return builder(BlobId.of(bucket, name)); } /** @@ -623,11 +690,11 @@ public static Builder builder(BucketInfo bucketInfo, String name, Long generatio * Returns a {@code BlobInfo} builder where blob identity is set using the provided values. */ public static Builder builder(String bucket, String name, Long generation) { - return new Builder().blobId(BlobId.of(bucket, name, generation)); + return builder(BlobId.of(bucket, name, generation)); } public static Builder builder(BlobId blobId) { - return new Builder().blobId(blobId); + return new BuilderImpl(blobId); } static BlobInfo fromPb(StorageObject storageObject) { diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java index 121f2eb63589..52b8c39321da 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BlobReadChannel.java @@ -29,6 +29,7 @@ import java.io.IOException; import java.io.Serializable; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Map; import java.util.Objects; import java.util.concurrent.Callable; @@ -55,7 +56,7 @@ class BlobReadChannel implements ReadChannel { private byte[] buffer; BlobReadChannel(StorageOptions serviceOptions, BlobId blob, - Map requestOptions) { + Map requestOptions) { this.serviceOptions = serviceOptions; this.blob = blob; this.requestOptions = requestOptions; @@ -91,9 +92,9 @@ public void close() { } } - private void validateOpen() throws IOException { + private void validateOpen() throws ClosedChannelException { if (!isOpen) { - throw new IOException("stream is closed"); + throw new ClosedChannelException(); } } @@ -126,7 +127,7 @@ public Tuple call() { return storageRpc.read(storageObject, requestOptions, position, toRead); } }, serviceOptions.retryParams(), StorageImpl.EXCEPTION_HANDLER); - if (lastEtag != null && !Objects.equals(result.x(), lastEtag)) { + if (result.y().length > 0 && lastEtag != null && !Objects.equals(result.x(), lastEtag)) { StringBuilder messageBuilder = new StringBuilder(); messageBuilder.append("Blob ").append(blob).append(" was updated while reading"); throw new StorageException(0, messageBuilder.toString()); diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java index 3acd3f5d79b9..d318626f4207 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Bucket.java @@ -23,13 +23,11 @@ import com.google.common.base.Function; import com.google.common.base.MoreObjects; -import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; import com.google.gcloud.Page; -import com.google.gcloud.PageImpl; import com.google.gcloud.spi.StorageRpc; import com.google.gcloud.storage.Storage.BlobGetOption; -import com.google.gcloud.storage.Storage.BlobTargetOption; -import com.google.gcloud.storage.Storage.BlobWriteOption; import com.google.gcloud.storage.Storage.BucketTargetOption; import java.io.IOException; @@ -39,87 +37,25 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; -import java.util.Iterator; import java.util.List; import java.util.Objects; +import java.util.Set; /** * A Google cloud storage bucket. * *

    Objects of this class are immutable. Operations that modify the bucket like {@link #update} * return a new object. To get a {@code Bucket} object with the most recent information use - * {@link #reload}. + * {@link #reload}. {@code Bucket} adds a layer of service-related functionality over + * {@link BucketInfo}. *

    */ -public final class Bucket { +public final class Bucket extends BucketInfo { - private final Storage storage; - private final BucketInfo info; + private static final long serialVersionUID = 8574601739542252586L; - private static class BlobPageFetcher implements PageImpl.NextPageFetcher { - - private static final long serialVersionUID = 3221100177471323801L; - - private final StorageOptions options; - private final Page infoPage; - - BlobPageFetcher(StorageOptions options, Page infoPage) { - this.options = options; - this.infoPage = infoPage; - } - - @Override - public Page nextPage() { - Page nextInfoPage = infoPage.nextPage(); - return new PageImpl<>(new BlobPageFetcher(options, nextInfoPage), - nextInfoPage.nextPageCursor(), new LazyBlobIterable(options, nextInfoPage.values())); - } - } - - private static class LazyBlobIterable implements Iterable, Serializable { - - private static final long serialVersionUID = -3092290247725378832L; - - private final StorageOptions options; - private final Iterable infoIterable; - private transient Storage storage; - - public LazyBlobIterable(StorageOptions options, Iterable infoIterable) { - this.options = options; - this.infoIterable = infoIterable; - this.storage = options.service(); - } - - private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { - in.defaultReadObject(); - this.storage = options.service(); - } - - @Override - public Iterator iterator() { - return Iterators.transform(infoIterable.iterator(), new Function() { - @Override - public Blob apply(BlobInfo blobInfo) { - return new Blob(storage, blobInfo); - } - }); - } - - @Override - public int hashCode() { - return Objects.hash(options, infoIterable); - } - - @Override - public boolean equals(Object obj) { - if (!(obj instanceof LazyBlobIterable)) { - return false; - } - LazyBlobIterable other = (LazyBlobIterable) obj; - return Objects.equals(options, other.options) - && Objects.equals(infoIterable, other.infoIterable); - } - } + private final StorageOptions options; + private transient Storage storage; /** * Class for specifying bucket source options when {@code Bucket} methods are used. @@ -132,7 +68,7 @@ private BucketSourceOption(StorageRpc.Option rpcOption) { super(rpcOption, null); } - private Storage.BucketSourceOption toSourceOptions(BucketInfo bucketInfo) { + private Storage.BucketSourceOption toSourceOption(BucketInfo bucketInfo) { switch (rpcOption()) { case IF_METAGENERATION_MATCH: return Storage.BucketSourceOption.metagenerationMatch(bucketInfo.metageneration()); @@ -176,7 +112,7 @@ static Storage.BucketSourceOption[] toSourceOptions(BucketInfo bucketInfo, new Storage.BucketSourceOption[options.length]; int index = 0; for (BucketSourceOption option : options) { - convertedOptions[index++] = option.toSourceOptions(bucketInfo); + convertedOptions[index++] = option.toSourceOption(bucketInfo); } return convertedOptions; } @@ -193,37 +129,404 @@ static Storage.BucketGetOption[] toGetOptions(BucketInfo bucketInfo, } /** - * Constructs a {@code Bucket} object for the provided {@code BucketInfo}. The storage service is - * used to issue requests. - * - * @param storage the storage service used for issuing requests - * @param info bucket's info + * Class for specifying blob target options when {@code Bucket} methods are used. */ - public Bucket(Storage storage, BucketInfo info) { - this.storage = checkNotNull(storage); - this.info = checkNotNull(info); + public static class BlobTargetOption extends Option { + + private static final Function TO_ENUM = + new Function() { + @Override + public StorageRpc.Option apply(BlobTargetOption blobTargetOption) { + return blobTargetOption.rpcOption(); + } + }; + private static final long serialVersionUID = 8345296337342509425L; + + private BlobTargetOption(StorageRpc.Option rpcOption, Object value) { + super(rpcOption, value); + } + + private StorageRpc.Tuple toTargetOption(BlobInfo blobInfo) { + BlobId blobId = blobInfo.blobId(); + switch (rpcOption()) { + case PREDEFINED_ACL: + return StorageRpc.Tuple.of(blobInfo, + Storage.BlobTargetOption.predefinedAcl((Storage.PredefinedAcl) value())); + case IF_GENERATION_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value()); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobTargetOption.generationMatch()); + case IF_GENERATION_NOT_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value()); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobTargetOption.generationNotMatch()); + case IF_METAGENERATION_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value()).build(), + Storage.BlobTargetOption.metagenerationMatch()); + case IF_METAGENERATION_NOT_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value()).build(), + Storage.BlobTargetOption.metagenerationNotMatch()); + default: + throw new AssertionError("Unexpected enum value"); + } + } + + /** + * Returns an option for specifying blob's predefined ACL configuration. + */ + public static BlobTargetOption predefinedAcl(Storage.PredefinedAcl acl) { + return new BlobTargetOption(StorageRpc.Option.PREDEFINED_ACL, acl); + } + + /** + * Returns an option that causes an operation to succeed only if the target blob does not exist. + * This option can not be provided together with {@link #generationMatch(long)} or + * {@link #generationNotMatch(long)}. + */ + public static BlobTargetOption doesNotExist() { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_MATCH, 0L); + } + + /** + * Returns an option for blob's data generation match. If this option is used the request will + * fail if generation does not match the provided value. This option can not be provided + * together with {@link #generationNotMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobTargetOption generationMatch(long generation) { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_MATCH, generation); + } + + /** + * Returns an option for blob's data generation mismatch. If this option is used the request + * will fail if blob's generation matches the provided value. This option can not be provided + * together with {@link #generationMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobTargetOption generationNotMatch(long generation) { + return new BlobTargetOption(StorageRpc.Option.IF_GENERATION_NOT_MATCH, generation); + } + + /** + * Returns an option for blob's metageneration match. If this option is used the request will + * fail if metageneration does not match the provided value. This option can not be provided + * together with {@link #metagenerationNotMatch(long)}. + */ + public static BlobTargetOption metagenerationMatch(long metageneration) { + return new BlobTargetOption(StorageRpc.Option.IF_METAGENERATION_MATCH, metageneration); + } + + /** + * Returns an option for blob's metageneration mismatch. If this option is used the request will + * fail if metageneration matches the provided value. This option can not be provided together + * with {@link #metagenerationMatch(long)}. + */ + public static BlobTargetOption metagenerationNotMatch(long metageneration) { + return new BlobTargetOption(StorageRpc.Option.IF_METAGENERATION_NOT_MATCH, metageneration); + } + + static StorageRpc.Tuple toTargetOptions( + BlobInfo info, BlobTargetOption... options) { + Set optionSet = + Sets.immutableEnumSet(Lists.transform(Arrays.asList(options), TO_ENUM)); + checkArgument(!(optionSet.contains(StorageRpc.Option.IF_METAGENERATION_NOT_MATCH) + && optionSet.contains(StorageRpc.Option.IF_METAGENERATION_MATCH)), + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + checkArgument(!(optionSet.contains(StorageRpc.Option.IF_GENERATION_NOT_MATCH) + && optionSet.contains(StorageRpc.Option.IF_GENERATION_MATCH)), + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + Storage.BlobTargetOption[] convertedOptions = new Storage.BlobTargetOption[options.length]; + BlobInfo targetInfo = info; + int index = 0; + for (BlobTargetOption option : options) { + StorageRpc.Tuple target = + option.toTargetOption(targetInfo); + targetInfo = target.x(); + convertedOptions[index++] = target.y(); + } + return StorageRpc.Tuple.of(targetInfo, convertedOptions); + } } /** - * Creates a {@code Bucket} object for the provided bucket name. Performs an RPC call to get the - * latest bucket information. - * - * @param storage the storage service used for issuing requests - * @param bucket bucket's name - * @param options blob get options - * @return the {@code Bucket} object or {@code null} if not found - * @throws StorageException upon failure + * Class for specifying blob write options when {@code Bucket} methods are used. */ - public static Bucket get(Storage storage, String bucket, Storage.BucketGetOption... options) { - BucketInfo info = storage.get(bucket, options); - return info != null ? new Bucket(storage, info) : null; + public static class BlobWriteOption implements Serializable { + + private static final Function TO_ENUM = + new Function() { + @Override + public Storage.BlobWriteOption.Option apply(BlobWriteOption blobWriteOption) { + return blobWriteOption.option; + } + }; + private static final long serialVersionUID = 4722190734541993114L; + + private final Storage.BlobWriteOption.Option option; + private final Object value; + + private StorageRpc.Tuple toWriteOption(BlobInfo blobInfo) { + BlobId blobId = blobInfo.blobId(); + switch (option) { + case PREDEFINED_ACL: + return StorageRpc.Tuple.of(blobInfo, + Storage.BlobWriteOption.predefinedAcl((Storage.PredefinedAcl) value)); + case IF_GENERATION_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobWriteOption.generationMatch()); + case IF_GENERATION_NOT_MATCH: + blobId = BlobId.of(blobId.bucket(), blobId.name(), (Long) value); + return StorageRpc.Tuple.of(blobInfo.toBuilder().blobId(blobId).build(), + Storage.BlobWriteOption.generationNotMatch()); + case IF_METAGENERATION_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value).build(), + Storage.BlobWriteOption.metagenerationMatch()); + case IF_METAGENERATION_NOT_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().metageneration((Long) value).build(), + Storage.BlobWriteOption.metagenerationNotMatch()); + case IF_MD5_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().md5((String) value).build(), + Storage.BlobWriteOption.md5Match()); + case IF_CRC32C_MATCH: + return StorageRpc.Tuple.of(blobInfo.toBuilder().crc32c((String) value).build(), + Storage.BlobWriteOption.crc32cMatch()); + default: + throw new AssertionError("Unexpected enum value"); + } + } + + private BlobWriteOption(Storage.BlobWriteOption.Option option, Object value) { + this.option = option; + this.value = value; + } + + @Override + public int hashCode() { + return Objects.hash(option, value); + } + + @Override + public boolean equals(Object obj) { + if (obj == null) { + return false; + } + if (!(obj instanceof BlobWriteOption)) { + return false; + } + final BlobWriteOption other = (BlobWriteOption) obj; + return this.option == other.option && Objects.equals(this.value, other.value); + } + + /** + * Returns an option for specifying blob's predefined ACL configuration. + */ + public static BlobWriteOption predefinedAcl(Storage.PredefinedAcl acl) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.PREDEFINED_ACL, acl); + } + + /** + * Returns an option that causes an operation to succeed only if the target blob does not exist. + * This option can not be provided together with {@link #generationMatch(long)} or + * {@link #generationNotMatch(long)}. + */ + public static BlobWriteOption doesNotExist() { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH, 0L); + } + + /** + * Returns an option for blob's data generation match. If this option is used the request will + * fail if generation does not match the provided value. This option can not be provided + * together with {@link #generationNotMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobWriteOption generationMatch(long generation) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH, generation); + } + + /** + * Returns an option for blob's data generation mismatch. If this option is used the request + * will fail if generation matches the provided value. This option can not be provided + * together with {@link #generationMatch(long)} or {@link #doesNotExist()}. + */ + public static BlobWriteOption generationNotMatch(long generation) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_GENERATION_NOT_MATCH, + generation); + } + + /** + * Returns an option for blob's metageneration match. If this option is used the request will + * fail if metageneration does not match the provided value. This option can not be provided + * together with {@link #metagenerationNotMatch(long)}. + */ + public static BlobWriteOption metagenerationMatch(long metageneration) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_METAGENERATION_MATCH, + metageneration); + } + + /** + * Returns an option for blob's metageneration mismatch. If this option is used the request will + * fail if metageneration matches the provided value. This option can not be provided together + * with {@link #metagenerationMatch(long)}. + */ + public static BlobWriteOption metagenerationNotMatch(long metageneration) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_METAGENERATION_NOT_MATCH, + metageneration); + } + + /** + * Returns an option for blob's data MD5 hash match. If this option is used the request will + * fail if blobs' data MD5 hash does not match the provided value. + */ + public static BlobWriteOption md5Match(String md5) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_MD5_MATCH, md5); + } + + /** + * Returns an option for blob's data CRC32C checksum match. If this option is used the request + * will fail if blobs' data CRC32C checksum does not match the provided value. + */ + public static BlobWriteOption crc32cMatch(String crc32c) { + return new BlobWriteOption(Storage.BlobWriteOption.Option.IF_CRC32C_MATCH, crc32c); + } + + static StorageRpc.Tuple toWriteOptions( + BlobInfo info, BlobWriteOption... options) { + Set optionSet = + Sets.immutableEnumSet(Lists.transform(Arrays.asList(options), TO_ENUM)); + checkArgument(!(optionSet.contains(Storage.BlobWriteOption.Option.IF_METAGENERATION_NOT_MATCH) + && optionSet.contains(Storage.BlobWriteOption.Option.IF_METAGENERATION_MATCH)), + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + checkArgument(!(optionSet.contains(Storage.BlobWriteOption.Option.IF_GENERATION_NOT_MATCH) + && optionSet.contains(Storage.BlobWriteOption.Option.IF_GENERATION_MATCH)), + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + Storage.BlobWriteOption[] convertedOptions = new Storage.BlobWriteOption[options.length]; + BlobInfo writeInfo = info; + int index = 0; + for (BlobWriteOption option : options) { + StorageRpc.Tuple write = option.toWriteOption(writeInfo); + writeInfo = write.x(); + convertedOptions[index++] = write.y(); + } + return StorageRpc.Tuple.of(writeInfo, convertedOptions); + } } /** - * Returns the bucket's information. + * Builder for {@code Bucket}. */ - public BucketInfo info() { - return info; + public static class Builder extends BucketInfo.Builder { + private final Storage storage; + private final BucketInfo.BuilderImpl infoBuilder; + + Builder(Bucket bucket) { + this.storage = bucket.storage; + this.infoBuilder = new BucketInfo.BuilderImpl(bucket); + } + + @Override + public Builder name(String name) { + infoBuilder.name(name); + return this; + } + + @Override + Builder id(String id) { + infoBuilder.id(id); + return this; + } + + @Override + Builder owner(Acl.Entity owner) { + infoBuilder.owner(owner); + return this; + } + + @Override + Builder selfLink(String selfLink) { + infoBuilder.selfLink(selfLink); + return this; + } + + @Override + public Builder versioningEnabled(Boolean enable) { + infoBuilder.versioningEnabled(enable); + return this; + } + + @Override + public Builder indexPage(String indexPage) { + infoBuilder.indexPage(indexPage); + return this; + } + + @Override + public Builder notFoundPage(String notFoundPage) { + infoBuilder.notFoundPage(notFoundPage); + return this; + } + + @Override + public Builder deleteRules(Iterable rules) { + infoBuilder.deleteRules(rules); + return this; + } + + @Override + public Builder storageClass(String storageClass) { + infoBuilder.storageClass(storageClass); + return this; + } + + @Override + public Builder location(String location) { + infoBuilder.location(location); + return this; + } + + @Override + Builder etag(String etag) { + infoBuilder.etag(etag); + return this; + } + + @Override + Builder createTime(Long createTime) { + infoBuilder.createTime(createTime); + return this; + } + + @Override + Builder metageneration(Long metageneration) { + infoBuilder.metageneration(metageneration); + return this; + } + + @Override + public Builder cors(Iterable cors) { + infoBuilder.cors(cors); + return this; + } + + @Override + public Builder acl(Iterable acl) { + infoBuilder.acl(acl); + return this; + } + + @Override + public Builder defaultAcl(Iterable acl) { + infoBuilder.defaultAcl(acl); + return this; + } + + @Override + public Bucket build() { + return new Bucket(storage, infoBuilder); + } + } + + Bucket(Storage storage, BucketInfo.BuilderImpl infoBuilder) { + super(infoBuilder); + this.storage = checkNotNull(storage); + this.options = storage.options(); } /** @@ -234,9 +537,9 @@ public BucketInfo info() { */ public boolean exists(BucketSourceOption... options) { int length = options.length; - Storage.BucketGetOption[] getOptions = Arrays.copyOf(toGetOptions(info, options), length + 1); + Storage.BucketGetOption[] getOptions = Arrays.copyOf(toGetOptions(this, options), length + 1); getOptions[length] = Storage.BucketGetOption.fields(); - return storage.get(info.name(), getOptions) != null; + return storage.get(name(), getOptions) != null; } /** @@ -247,7 +550,7 @@ public boolean exists(BucketSourceOption... options) { * @throws StorageException upon failure */ public Bucket reload(BucketSourceOption... options) { - return Bucket.get(storage, info.name(), toGetOptions(info, options)); + return storage.get(name(), toGetOptions(this, options)); } /** @@ -255,16 +558,14 @@ public Bucket reload(BucketSourceOption... options) { * is returned. By default no checks are made on the metadata generation of the current bucket. * If you want to update the information only if the current bucket metadata are at their latest * version use the {@code metagenerationMatch} option: - * {@code bucket.update(newInfo, BucketTargetOption.metagenerationMatch())} + * {@code bucket.update(BucketTargetOption.metagenerationMatch())} * - * @param bucketInfo new bucket's information. Name must match the one of the current bucket * @param options update options * @return a {@code Bucket} object with updated information * @throws StorageException upon failure */ - public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { - checkArgument(Objects.equals(bucketInfo.name(), info.name()), "Bucket name must match"); - return new Bucket(storage, storage.update(bucketInfo, options)); + public Bucket update(BucketTargetOption... options) { + return storage.update(this, options); } /** @@ -275,36 +576,33 @@ public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { * @throws StorageException upon failure */ public boolean delete(BucketSourceOption... options) { - return storage.delete(info.name(), toSourceOptions(info, options)); + return storage.delete(name(), toSourceOptions(this, options)); } /** * Returns the paginated list of {@code Blob} in this bucket. - * + * * @param options options for listing blobs * @throws StorageException upon failure */ public Page list(Storage.BlobListOption... options) { - Page infoPage = storage.list(info.name(), options); - StorageOptions storageOptions = storage.options(); - return new PageImpl<>(new BlobPageFetcher(storageOptions, infoPage), infoPage.nextPageCursor(), - new LazyBlobIterable(storageOptions, infoPage.values())); + return storage.list(name(), options); } /** * Returns the requested blob in this bucket or {@code null} if not found. - * + * * @param blob name of the requested blob * @param options blob search options * @throws StorageException upon failure */ public Blob get(String blob, BlobGetOption... options) { - return new Blob(storage, storage.get(BlobId.of(info.name(), blob), options)); + return storage.get(BlobId.of(name(), blob), options); } /** * Returns a list of requested blobs in this bucket. Blobs that do not exist are null. - * + * * @param blobName1 first blob to get * @param blobName2 second blob to get * @param blobNames other blobs to get @@ -313,16 +611,16 @@ public Blob get(String blob, BlobGetOption... options) { */ public List get(String blobName1, String blobName2, String... blobNames) { BatchRequest.Builder batch = BatchRequest.builder(); - batch.get(info.name(), blobName1); - batch.get(info.name(), blobName2); + batch.get(name(), blobName1); + batch.get(name(), blobName2); for (String name : blobNames) { - batch.get(info.name(), name); + batch.get(name(), name); } List blobs = new ArrayList<>(blobNames.length); BatchResponse response = storage.submit(batch.build()); - for (BatchResponse.Result result : response.gets()) { + for (BatchResponse.Result result : response.gets()) { BlobInfo blobInfo = result.get(); - blobs.add(blobInfo != null ? new Blob(storage, blobInfo) : null); + blobs.add(blobInfo != null ? new Blob(storage, new BlobInfo.BuilderImpl(blobInfo)) : null); } return Collections.unmodifiableList(blobs); } @@ -332,7 +630,7 @@ public List get(String blobName1, String blobName2, String... blobNames) { * For large content, {@link Blob#writer(com.google.gcloud.storage.Storage.BlobWriteOption...)} * is recommended as it uses resumable upload. MD5 and CRC32C hashes of {@code content} are * computed and used for validating transferred data. - * + * * @param blob a blob name * @param content the blob content * @param contentType the blob content type. If {@code null} then @@ -342,16 +640,18 @@ public List get(String blobName1, String blobName2, String... blobNames) { * @throws StorageException upon failure */ public Blob create(String blob, byte[] content, String contentType, BlobTargetOption... options) { - BlobInfo blobInfo = BlobInfo.builder(BlobId.of(info.name(), blob)) + BlobInfo blobInfo = BlobInfo.builder(BlobId.of(name(), blob)) .contentType(MoreObjects.firstNonNull(contentType, Storage.DEFAULT_CONTENT_TYPE)).build(); - return new Blob(storage, storage.create(blobInfo, content, options)); + StorageRpc.Tuple target = + BlobTargetOption.toTargetOptions(blobInfo, options); + return storage.create(target.x(), content, target.y()); } /** * Creates a new blob in this bucket. Direct upload is used to upload {@code content}. * For large content, {@link Blob#writer(com.google.gcloud.storage.Storage.BlobWriteOption...)} * is recommended as it uses resumable upload. - * + * * @param blob a blob name * @param content the blob content as a stream * @param contentType the blob content type. If {@code null} then @@ -362,9 +662,11 @@ public Blob create(String blob, byte[] content, String contentType, BlobTargetOp */ public Blob create(String blob, InputStream content, String contentType, BlobWriteOption... options) { - BlobInfo blobInfo = BlobInfo.builder(BlobId.of(info.name(), blob)) + BlobInfo blobInfo = BlobInfo.builder(BlobId.of(name(), blob)) .contentType(MoreObjects.firstNonNull(contentType, Storage.DEFAULT_CONTENT_TYPE)).build(); - return new Blob(storage, storage.create(blobInfo, content, options)); + StorageRpc.Tuple write = + BlobWriteOption.toWriteOptions(blobInfo, options); + return storage.create(write.x(), content, write.y()); } /** @@ -373,4 +675,29 @@ public Blob create(String blob, InputStream content, String contentType, public Storage storage() { return storage; } + + @Override + public Builder toBuilder() { + return new Builder(this); + } + + @Override + public boolean equals(Object obj) { + return obj instanceof Bucket && Objects.equals(toPb(), ((Bucket) obj).toPb()) + && Objects.equals(options, ((Bucket) obj).options); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), options); + } + + private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { + in.defaultReadObject(); + this.storage = options.service(); + } + + static Bucket fromPb(Storage storage, com.google.api.services.storage.model.Bucket bucketPb) { + return new Bucket(storage, new BucketInfo.BuilderImpl(BucketInfo.fromPb(bucketPb))); + } } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java index 62fbf9c6521f..a1de1a07e03e 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/BucketInfo.java @@ -16,8 +16,8 @@ package com.google.gcloud.storage; -import static com.google.api.client.repackaged.com.google.common.base.Preconditions.checkNotNull; import static com.google.common.base.MoreObjects.firstNonNull; +import static com.google.common.base.Preconditions.checkNotNull; import static com.google.common.collect.Lists.transform; import com.google.api.client.json.jackson2.JacksonFactory; @@ -48,7 +48,7 @@ * @see Concepts and * Terminology */ -public final class BucketInfo implements Serializable { +public class BucketInfo implements Serializable { static final Function FROM_PB_FUNCTION = new Function() { @@ -317,7 +317,99 @@ void populateCondition(Rule.Condition condition) { } } - public static final class Builder { + /** + * Builder for {@code BucketInfo}. + */ + public abstract static class Builder { + /** + * Sets the bucket's name. + */ + public abstract Builder name(String name); + + abstract Builder id(String id); + + abstract Builder owner(Acl.Entity owner); + + abstract Builder selfLink(String selfLink); + + /** + * Sets whether versioning should be enabled for this bucket. When set to true, versioning is + * fully enabled. + */ + public abstract Builder versioningEnabled(Boolean enable); + + /** + * Sets the bucket's website index page. Behaves as the bucket's directory index where missing + * blobs are treated as potential directories. + */ + public abstract Builder indexPage(String indexPage); + + /** + * Sets the custom object to return when a requested resource is not found. + */ + public abstract Builder notFoundPage(String notFoundPage); + + /** + * Sets the bucket's lifecycle configuration as a number of delete rules. + * + * @see Lifecycle Management + */ + public abstract Builder deleteRules(Iterable rules); + + /** + * Sets the bucket's storage class. This defines how blobs in the bucket are stored and + * determines the SLA and the cost of storage. A list of supported values is available + * here. + */ + public abstract Builder storageClass(String storageClass); + + /** + * Sets the bucket's location. Data for blobs in the bucket resides in physical storage within + * this region. A list of supported values is available + * here. + */ + public abstract Builder location(String location); + + abstract Builder etag(String etag); + + abstract Builder createTime(Long createTime); + + abstract Builder metageneration(Long metageneration); + + /** + * Sets the bucket's Cross-Origin Resource Sharing (CORS) configuration. + * + * @see + * Cross-Origin Resource Sharing (CORS) + */ + public abstract Builder cors(Iterable cors); + + /** + * Sets the bucket's access control configuration. + * + * @see + * About Access Control Lists + */ + public abstract Builder acl(Iterable acl); + + /** + * Sets the default access control configuration to apply to bucket's blobs when no other + * configuration is specified. + * + * @see + * About Access Control Lists + */ + public abstract Builder defaultAcl(Iterable acl); + + /** + * Creates a {@code BucketInfo} object. + */ + public abstract BucketInfo build(); + } + + static final class BuilderImpl extends Builder { private String id; private String name; @@ -336,9 +428,11 @@ public static final class Builder { private List acl; private List defaultAcl; - private Builder() {} + BuilderImpl(String name) { + this.name = name; + } - private Builder(BucketInfo bucketInfo) { + BuilderImpl(BucketInfo bucketInfo) { id = bucketInfo.id; name = bucketInfo.name; etag = bucketInfo.etag; @@ -357,144 +451,110 @@ private Builder(BucketInfo bucketInfo) { deleteRules = bucketInfo.deleteRules; } - /** - * Sets the bucket's name. - */ + @Override public Builder name(String name) { this.name = checkNotNull(name); return this; } + @Override Builder id(String id) { this.id = id; return this; } + @Override Builder owner(Acl.Entity owner) { this.owner = owner; return this; } + @Override Builder selfLink(String selfLink) { this.selfLink = selfLink; return this; } - /** - * Sets whether versioning should be enabled for this bucket. When set to true, versioning is - * fully enabled. - */ + @Override public Builder versioningEnabled(Boolean enable) { this.versioningEnabled = firstNonNull(enable, Data.nullOf(Boolean.class)); return this; } - /** - * Sets the bucket's website index page. Behaves as the bucket's directory index where missing - * blobs are treated as potential directories. - */ + @Override public Builder indexPage(String indexPage) { this.indexPage = indexPage; return this; } - /** - * Sets the custom object to return when a requested resource is not found. - */ + @Override public Builder notFoundPage(String notFoundPage) { this.notFoundPage = notFoundPage; return this; } - /** - * Sets the bucket's lifecycle configuration as a number of delete rules. - * - * @see Lifecycle Management - */ + @Override public Builder deleteRules(Iterable rules) { this.deleteRules = rules != null ? ImmutableList.copyOf(rules) : null; return this; } - /** - * Sets the bucket's storage class. This defines how blobs in the bucket are stored and - * determines the SLA and the cost of storage. A list of supported values is available - * here. - */ + @Override public Builder storageClass(String storageClass) { this.storageClass = storageClass; return this; } - /** - * Sets the bucket's location. Data for blobs in the bucket resides in physical storage within - * this region. A list of supported values is available - * here. - */ + @Override public Builder location(String location) { this.location = location; return this; } + @Override Builder etag(String etag) { this.etag = etag; return this; } + @Override Builder createTime(Long createTime) { this.createTime = createTime; return this; } + @Override Builder metageneration(Long metageneration) { this.metageneration = metageneration; return this; } - /** - * Sets the bucket's Cross-Origin Resource Sharing (CORS) configuration. - * - * @see - * Cross-Origin Resource Sharing (CORS) - */ + @Override public Builder cors(Iterable cors) { this.cors = cors != null ? ImmutableList.copyOf(cors) : null; return this; } - /** - * Sets the bucket's access control configuration. - * - * @see - * About Access Control Lists - */ + @Override public Builder acl(Iterable acl) { this.acl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } - /** - * Sets the default access control configuration to apply to bucket's blobs when no other - * configuration is specified. - * - * @see - * About Access Control Lists - */ + @Override public Builder defaultAcl(Iterable acl) { this.defaultAcl = acl != null ? ImmutableList.copyOf(acl) : null; return this; } - /** - * Creates a {@code BucketInfo} object. - */ + @Override public BucketInfo build() { checkNotNull(name); return new BucketInfo(this); } } - private BucketInfo(Builder builder) { + BucketInfo(BuilderImpl builder) { id = builder.id; name = builder.name; etag = builder.etag; @@ -649,7 +709,7 @@ public List defaultAcl() { * Returns a builder for the current bucket. */ public Builder toBuilder() { - return new Builder(this); + return new BuilderImpl(this); } @Override @@ -659,7 +719,8 @@ public int hashCode() { @Override public boolean equals(Object obj) { - return obj instanceof BucketInfo && Objects.equals(toPb(), ((BucketInfo) obj).toPb()); + return obj != null && obj.getClass().equals(BucketInfo.class) + && Objects.equals(toPb(), ((BucketInfo) obj).toPb()); } @Override @@ -743,11 +804,11 @@ public static BucketInfo of(String name) { * Returns a {@code BucketInfo} builder where the bucket's name is set to the provided name. */ public static Builder builder(String name) { - return new Builder().name(name); + return new BuilderImpl(name); } static BucketInfo fromPb(com.google.api.services.storage.model.Bucket bucketPb) { - Builder builder = new Builder().name(bucketPb.getName()); + Builder builder = new BuilderImpl(bucketPb.getName()); if (bucketPb.getId() != null) { builder.id(bucketPb.getId()); } diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java index 1e5427a847d4..7eb91d0910a2 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/CopyWriter.java @@ -57,9 +57,10 @@ public class CopyWriter implements Restorable { * is {@code false} will block until all pending chunks are copied. * *

    This method has the same effect of doing: - *

        {@code while (!copyWriter.isDone()) {
    -   *        copyWriter.copyChunk();
    -   *    }}
    +   * 
     {@code
    +   * while (!copyWriter.isDone()) {
    +   *    copyWriter.copyChunk();
    +   * }}
        * 
    * * @throws StorageException upon failure diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java index b550015d0516..065bcca7c9e8 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/Storage.java @@ -655,7 +655,7 @@ public static BucketListOption prefix(String prefix) { */ public static BucketListOption fields(BucketField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("items(").append(BucketField.selector(fields)).append(")"); + builder.append("items(").append(BucketField.selector(fields)).append("),nextPageToken"); return new BucketListOption(StorageRpc.Option.FIELDS, builder.toString()); } } @@ -700,6 +700,15 @@ public static BlobListOption recursive(boolean recursive) { return new BlobListOption(StorageRpc.Option.DELIMITER, recursive); } + /** + * If set to {@code true}, lists all versions of a blob. The default is {@code false}. + * + * @see Object Versioning + */ + public static BlobListOption versions(boolean versions) { + return new BlobListOption(StorageRpc.Option.VERSIONS, versions); + } + /** * Returns an option to specify the blob's fields to be returned by the RPC call. If this option * is not provided all blob's fields are returned. {@code BlobListOption.fields}) can be used to @@ -708,7 +717,7 @@ public static BlobListOption recursive(boolean recursive) { */ public static BlobListOption fields(BlobField... fields) { StringBuilder builder = new StringBuilder(); - builder.append("items(").append(BlobField.selector(fields)).append(")"); + builder.append("items(").append(BlobField.selector(fields)).append("),nextPageToken"); return new BlobListOption(StorageRpc.Option.FIELDS, builder.toString()); } } @@ -1206,126 +1215,130 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx } /** - * Create a new bucket. + * Creates a new bucket. * - * @return a complete bucket information + * @return a complete bucket * @throws StorageException upon failure */ - BucketInfo create(BucketInfo bucketInfo, BucketTargetOption... options); + Bucket create(BucketInfo bucketInfo, BucketTargetOption... options); /** - * Create a new blob with no content. + * Creates a new blob with no content. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure */ - BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options); + Blob create(BlobInfo blobInfo, BlobTargetOption... options); /** - * Create a new blob. Direct upload is used to upload {@code content}. For large content, + * Creates a new blob. Direct upload is used to upload {@code content}. For large content, * {@link #writer} is recommended as it uses resumable upload. MD5 and CRC32C hashes of * {@code content} are computed and used for validating transferred data. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure * @see Hashes and ETags */ - BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options); + Blob create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options); /** - * Create a new blob. Direct upload is used to upload {@code content}. For large content, + * Creates a new blob. Direct upload is used to upload {@code content}. For large content, * {@link #writer} is recommended as it uses resumable upload. By default any md5 and crc32c * values in the given {@code blobInfo} are ignored unless requested via the * {@code BlobWriteOption.md5Match} and {@code BlobWriteOption.crc32cMatch} options. The given * input stream is closed upon success. * - * @return a complete blob information + * @return a [@code Blob} with complete information * @throws StorageException upon failure */ - BlobInfo create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options); + Blob create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options); /** - * Return the requested bucket or {@code null} if not found. + * Returns the requested bucket or {@code null} if not found. * * @throws StorageException upon failure */ - BucketInfo get(String bucket, BucketGetOption... options); + Bucket get(String bucket, BucketGetOption... options); /** - * Return the requested blob or {@code null} if not found. + * Returns the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(String bucket, String blob, BlobGetOption... options); + Blob get(String bucket, String blob, BlobGetOption... options); /** - * Return the requested blob or {@code null} if not found. + * Returns the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(BlobId blob, BlobGetOption... options); + Blob get(BlobId blob, BlobGetOption... options); /** - * Return the requested blob or {@code null} if not found. + * Returns the requested blob or {@code null} if not found. * * @throws StorageException upon failure */ - BlobInfo get(BlobId blob); + Blob get(BlobId blob); /** - * List the project's buckets. + * Lists the project's buckets. * * @throws StorageException upon failure */ - Page list(BucketListOption... options); + Page list(BucketListOption... options); /** - * List the bucket's blobs. + * Lists the bucket's blobs. * * @throws StorageException upon failure */ - Page list(String bucket, BlobListOption... options); + Page list(String bucket, BlobListOption... options); /** - * Update bucket information. + * Updates bucket information. * * @return the updated bucket * @throws StorageException upon failure */ - BucketInfo update(BucketInfo bucketInfo, BucketTargetOption... options); + Bucket update(BucketInfo bucketInfo, BucketTargetOption... options); /** - * Update blob information. Original metadata are merged with metadata in the provided + * Updates blob information. Original metadata are merged with metadata in the provided * {@code blobInfo}. To replace metadata instead you first have to unset them. Unsetting metadata * can be done by setting the provided {@code blobInfo}'s metadata to {@code null}. * *

    Example usage of replacing blob's metadata: - *

        {@code service.update(BlobInfo.builder("bucket", "name").metadata(null).build());}
    -   *    {@code service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());}
    +   * 
     {@code
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(null).build());
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());
    +   * }
        * 
    * * @return the updated blob * @throws StorageException upon failure */ - BlobInfo update(BlobInfo blobInfo, BlobTargetOption... options); + Blob update(BlobInfo blobInfo, BlobTargetOption... options); /** - * Update blob information. Original metadata are merged with metadata in the provided + * Updates blob information. Original metadata are merged with metadata in the provided * {@code blobInfo}. To replace metadata instead you first have to unset them. Unsetting metadata * can be done by setting the provided {@code blobInfo}'s metadata to {@code null}. * *

    Example usage of replacing blob's metadata: - *

        {@code service.update(BlobInfo.builder("bucket", "name").metadata(null).build());}
    -   *    {@code service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());}
    +   * 
     {@code
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(null).build());
    +   * service.update(BlobInfo.builder("bucket", "name").metadata(newMetadata).build());
    +   * }
        * 
    * * @return the updated blob * @throws StorageException upon failure */ - BlobInfo update(BlobInfo blobInfo); + Blob update(BlobInfo blobInfo); /** - * Delete the requested bucket. + * Deletes the requested bucket. * * @return {@code true} if bucket was deleted, {@code false} if it was not found * @throws StorageException upon failure @@ -1333,7 +1346,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx boolean delete(String bucket, BucketSourceOption... options); /** - * Delete the requested blob. + * Deletes the requested blob. * * @return {@code true} if blob was deleted, {@code false} if it was not found * @throws StorageException upon failure @@ -1341,7 +1354,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx boolean delete(String bucket, String blob, BlobSourceOption... options); /** - * Delete the requested blob. + * Deletes the requested blob. * * @return {@code true} if blob was deleted, {@code false} if it was not found * @throws StorageException upon failure @@ -1349,7 +1362,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx boolean delete(BlobId blob, BlobSourceOption... options); /** - * Delete the requested blob. + * Deletes the requested blob. * * @return {@code true} if blob was deleted, {@code false} if it was not found * @throws StorageException upon failure @@ -1357,12 +1370,12 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx boolean delete(BlobId blob); /** - * Send a compose request. + * Sends a compose request. * * @return the composed blob * @throws StorageException upon failure */ - BlobInfo compose(ComposeRequest composeRequest); + Blob compose(ComposeRequest composeRequest); /** * Sends a copy request. Returns a {@link CopyWriter} object for the provided @@ -1373,14 +1386,15 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * might issue multiple RPC calls depending on blob's size. * *

    Example usage of copy: - *

        {@code BlobInfo blob = service.copy(copyRequest).result();}
    +   * 
     {@code BlobInfo blob = service.copy(copyRequest).result();}
        * 
    * To explicitly issue chunk copy requests use {@link CopyWriter#copyChunk()} instead: - *
        {@code CopyWriter copyWriter = service.copy(copyRequest);
    -   *    while (!copyWriter.isDone()) {
    -   *        copyWriter.copyChunk();
    -   *    }
    -   *    BlobInfo blob = copyWriter.result();
    +   * 
     {@code
    +   * CopyWriter copyWriter = service.copy(copyRequest);
    +   * while (!copyWriter.isDone()) {
    +   *     copyWriter.copyChunk();
    +   * }
    +   * BlobInfo blob = copyWriter.result();
        * }
        * 
    * @@ -1408,7 +1422,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx byte[] readAllBytes(BlobId blob, BlobSourceOption... options); /** - * Send a batch request. + * Sends a batch request. * * @return the batch response * @throws StorageException upon failure @@ -1416,7 +1430,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx BatchResponse submit(BatchRequest batchRequest); /** - * Return a channel for reading the blob's content. The blob's latest generation is read. If the + * Returns a channel for reading the blob's content. The blob's latest generation is read. If the * blob changes while reading (i.e. {@link BlobInfo#etag()} changes), subsequent calls to * {@code blobReadChannel.read(ByteBuffer)} may throw {@link StorageException}. * @@ -1429,7 +1443,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx ReadChannel reader(String bucket, String blob, BlobSourceOption... options); /** - * Return a channel for reading the blob's content. If {@code blob.generation()} is set + * Returns a channel for reading the blob's content. If {@code blob.generation()} is set * data corresponding to that generation is read. If {@code blob.generation()} is {@code null} * the blob's latest generation is read. If the blob changes while reading (i.e. * {@link BlobInfo#etag()} changes), subsequent calls to {@code blobReadChannel.read(ByteBuffer)} @@ -1445,7 +1459,7 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx ReadChannel reader(BlobId blob, BlobSourceOption... options); /** - * Create a blob and return a channel for writing its content. By default any md5 and crc32c + * Creates a blob and return a channel for writing its content. By default any md5 and crc32c * values in the given {@code blobInfo} are ignored unless requested via the * {@code BlobWriteOption.md5Match} and {@code BlobWriteOption.crc32cMatch} options. * @@ -1462,12 +1476,12 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * accessible blobs, but don't want to require users to explicitly log in. * *

    Example usage of creating a signed URL that is valid for 2 weeks: - *

       {@code
    -   *     service.signUrl(BlobInfo.builder("bucket", "name").build(), 14, TimeUnit.DAYS);
    +   * 
     {@code
    +   * service.signUrl(BlobInfo.builder("bucket", "name").build(), 14, TimeUnit.DAYS);
        * }
    * * @param blobInfo the blob associated with the signed URL - * @param duration time until the signed URL expires, expressed in {@code unit}. The finer + * @param duration time until the signed URL expires, expressed in {@code unit}. The finest * granularity supported is 1 second, finer granularities will be truncated * @param unit time unit of the {@code duration} parameter * @param options optional URL signing options @@ -1479,11 +1493,11 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * Gets the requested blobs. A batch request is used to perform this call. * * @param blobIds blobs to get - * @return an immutable list of {@code BlobInfo} objects. If a blob does not exist or access to it + * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it * has been denied the corresponding item in the list is {@code null}. * @throws StorageException upon failure */ - List get(BlobId... blobIds); + List get(BlobId... blobIds); /** * Updates the requested blobs. A batch request is used to perform this call. Original metadata @@ -1493,11 +1507,11 @@ private static void checkContentType(BlobInfo blobInfo) throws IllegalArgumentEx * {@link #update(com.google.gcloud.storage.BlobInfo)} for a code example. * * @param blobInfos blobs to update - * @return an immutable list of {@code BlobInfo} objects. If a blob does not exist or access to it + * @return an immutable list of {@code Blob} objects. If a blob does not exist or access to it * has been denied the corresponding item in the list is {@code null}. * @throws StorageException upon failure */ - List update(BlobInfo... blobInfos); + List update(BlobInfo... blobInfos); /** * Deletes the requested blobs. A batch request is used to perform this call. diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java index 0c952c9a65d6..ee85b80d6e13 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageException.java @@ -33,7 +33,7 @@ */ public class StorageException extends BaseServiceException { - // see: https://cloud.google.com/storage/docs/concepts-techniques#practices + // see: https://cloud.google.com/storage/docs/resumable-uploads-xml#practices private static final Set RETRYABLE_ERRORS = ImmutableSet.of( new Error(504, null), new Error(503, null), diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java index b6a833f26ab4..de77cba021a1 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/StorageImpl.java @@ -33,7 +33,6 @@ import com.google.api.services.storage.model.StorageObject; import com.google.auth.oauth2.ServiceAccountCredentials; import com.google.common.base.Function; -import com.google.common.base.Functions; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Iterables; @@ -78,6 +77,14 @@ final class StorageImpl extends BaseService implements Storage { private static final String EMPTY_BYTE_ARRAY_MD5 = "1B2M2Y8AsgTpgAmY7PhCfg=="; private static final String EMPTY_BYTE_ARRAY_CRC32C = "AAAAAA=="; + private static final Function, Boolean> DELETE_FUNCTION = + new Function, Boolean>() { + @Override + public Boolean apply(Tuple tuple) { + return tuple.y(); + } + }; + private final StorageRpc storageRpc; StorageImpl(StorageOptions options) { @@ -86,11 +93,11 @@ final class StorageImpl extends BaseService implements Storage { } @Override - public BucketInfo create(BucketInfo bucketInfo, BucketTargetOption... options) { + public Bucket create(BucketInfo bucketInfo, BucketTargetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = bucketInfo.toPb(); final Map optionsMap = optionMap(bucketInfo, options); try { - return BucketInfo.fromPb(runWithRetries( + return Bucket.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.storage.model.Bucket call() { @@ -103,7 +110,7 @@ public com.google.api.services.storage.model.Bucket call() { } @Override - public BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options) { + public Blob create(BlobInfo blobInfo, BlobTargetOption... options) { BlobInfo updatedInfo = blobInfo.toBuilder() .md5(EMPTY_BYTE_ARRAY_MD5) .crc32c(EMPTY_BYTE_ARRAY_CRC32C) @@ -112,7 +119,7 @@ public BlobInfo create(BlobInfo blobInfo, BlobTargetOption... options) { } @Override - public BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options) { + public Blob create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options) { content = firstNonNull(content, EMPTY_BYTE_ARRAY); BlobInfo updatedInfo = blobInfo.toBuilder() .md5(BaseEncoding.base64().encode(Hashing.md5().hashBytes(content).asBytes())) @@ -123,16 +130,16 @@ public BlobInfo create(BlobInfo blobInfo, byte[] content, BlobTargetOption... op } @Override - public BlobInfo create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options) { + public Blob create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options) { Tuple targetOptions = BlobTargetOption.convert(blobInfo, options); return create(targetOptions.x(), content, targetOptions.y()); } - private BlobInfo create(BlobInfo info, final InputStream content, BlobTargetOption... options) { + private Blob create(BlobInfo info, final InputStream content, BlobTargetOption... options) { final StorageObject blobPb = info.toPb(); final Map optionsMap = optionMap(info, options); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.create(blobPb, @@ -145,7 +152,7 @@ public StorageObject call() { } @Override - public BucketInfo get(String bucket, BucketGetOption... options) { + public Bucket get(String bucket, BucketGetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = BucketInfo.of(bucket).toPb(); final Map optionsMap = optionMap(options); try { @@ -156,19 +163,19 @@ public com.google.api.services.storage.model.Bucket call() { return storageRpc.get(bucketPb, optionsMap); } }, options().retryParams(), EXCEPTION_HANDLER); - return answer == null ? null : BucketInfo.fromPb(answer); + return answer == null ? null : Bucket.fromPb(this, answer); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } @Override - public BlobInfo get(String bucket, String blob, BlobGetOption... options) { + public Blob get(String bucket, String blob, BlobGetOption... options) { return get(BlobId.of(bucket, blob), options); } @Override - public BlobInfo get(BlobId blob, BlobGetOption... options) { + public Blob get(BlobId blob, BlobGetOption... options) { final StorageObject storedObject = blob.toPb(); final Map optionsMap = optionMap(blob, options); try { @@ -178,18 +185,18 @@ public StorageObject call() { return storageRpc.get(storedObject, optionsMap); } }, options().retryParams(), EXCEPTION_HANDLER); - return storageObject == null ? null : BlobInfo.fromPb(storageObject); + return storageObject == null ? null : Blob.fromPb(this, storageObject); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } @Override - public BlobInfo get(BlobId blob) { + public Blob get(BlobId blob) { return get(blob, new BlobGetOption[0]); } - private static class BucketPageFetcher implements NextPageFetcher { + private static class BucketPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 5850406828803613729L; private final Map requestOptions; @@ -204,12 +211,12 @@ private static class BucketPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listBuckets(serviceOptions, requestOptions); } } - private static class BlobPageFetcher implements NextPageFetcher { + private static class BlobPageFetcher implements NextPageFetcher { private static final long serialVersionUID = 81807334445874098L; private final Map requestOptions; @@ -225,22 +232,22 @@ private static class BlobPageFetcher implements NextPageFetcher { } @Override - public Page nextPage() { + public Page nextPage() { return listBlobs(bucket, serviceOptions, requestOptions); } } @Override - public Page list(BucketListOption... options) { + public Page list(BucketListOption... options) { return listBuckets(options(), optionMap(options)); } @Override - public Page list(final String bucket, BlobListOption... options) { + public Page list(final String bucket, BlobListOption... options) { return listBlobs(bucket, options(), optionMap(options)); } - private static Page listBuckets(final StorageOptions serviceOptions, + private static Page listBuckets(final StorageOptions serviceOptions, final Map optionsMap) { try { Tuple> result = runWithRetries( @@ -251,22 +258,23 @@ public Tuple> cal } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable buckets = - result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), - new Function() { + Iterable buckets = + result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), + new Function() { @Override - public BucketInfo apply(com.google.api.services.storage.model.Bucket bucketPb) { - return BucketInfo.fromPb(bucketPb); + public Bucket apply(com.google.api.services.storage.model.Bucket bucketPb) { + return Bucket.fromPb(serviceOptions.service(), bucketPb); } }); - return new PageImpl<>(new BucketPageFetcher(serviceOptions, cursor, optionsMap), cursor, + return new PageImpl<>( + new BucketPageFetcher(serviceOptions, cursor, optionsMap), cursor, buckets); } catch (RetryHelperException e) { throw StorageException.translateAndThrow(e); } } - private static Page listBlobs(final String bucket, + private static Page listBlobs(final String bucket, final StorageOptions serviceOptions, final Map optionsMap) { try { Tuple> result = runWithRetries( @@ -277,15 +285,17 @@ public Tuple> call() { } }, serviceOptions.retryParams(), EXCEPTION_HANDLER); String cursor = result.x(); - Iterable blobs = - result.y() == null ? ImmutableList.of() : Iterables.transform(result.y(), - new Function() { + Iterable blobs = + result.y() == null + ? ImmutableList.of() + : Iterables.transform(result.y(), new Function() { @Override - public BlobInfo apply(StorageObject storageObject) { - return BlobInfo.fromPb(storageObject); + public Blob apply(StorageObject storageObject) { + return Blob.fromPb(serviceOptions.service(), storageObject); } }); - return new PageImpl<>(new BlobPageFetcher(bucket, serviceOptions, cursor, optionsMap), + return new PageImpl<>( + new BlobPageFetcher(bucket, serviceOptions, cursor, optionsMap), cursor, blobs); } catch (RetryHelperException e) { @@ -294,11 +304,11 @@ public BlobInfo apply(StorageObject storageObject) { } @Override - public BucketInfo update(BucketInfo bucketInfo, BucketTargetOption... options) { + public Bucket update(BucketInfo bucketInfo, BucketTargetOption... options) { final com.google.api.services.storage.model.Bucket bucketPb = bucketInfo.toPb(); final Map optionsMap = optionMap(bucketInfo, options); try { - return BucketInfo.fromPb(runWithRetries( + return Bucket.fromPb(this, runWithRetries( new Callable() { @Override public com.google.api.services.storage.model.Bucket call() { @@ -311,11 +321,11 @@ public com.google.api.services.storage.model.Bucket call() { } @Override - public BlobInfo update(BlobInfo blobInfo, BlobTargetOption... options) { + public Blob update(BlobInfo blobInfo, BlobTargetOption... options) { final StorageObject storageObject = blobInfo.toPb(); final Map optionsMap = optionMap(blobInfo, options); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.patch(storageObject, optionsMap); @@ -327,7 +337,7 @@ public StorageObject call() { } @Override - public BlobInfo update(BlobInfo blobInfo) { + public Blob update(BlobInfo blobInfo) { return update(blobInfo, new BlobTargetOption[0]); } @@ -374,7 +384,7 @@ public boolean delete(BlobId blob) { } @Override - public BlobInfo compose(final ComposeRequest composeRequest) { + public Blob compose(final ComposeRequest composeRequest) { final List sources = Lists.newArrayListWithCapacity(composeRequest.sourceBlobs().size()); for (ComposeRequest.SourceBlob sourceBlob : composeRequest.sourceBlobs()) { @@ -386,7 +396,7 @@ public BlobInfo compose(final ComposeRequest composeRequest) { final Map targetOptions = optionMap(composeRequest.target().generation(), composeRequest.target().metageneration(), composeRequest.targetOptions()); try { - return BlobInfo.fromPb(runWithRetries(new Callable() { + return Blob.fromPb(this, runWithRetries(new Callable() { @Override public StorageObject call() { return storageRpc.compose(sources, target, targetOptions); @@ -468,18 +478,19 @@ public BatchResponse submit(BatchRequest batchRequest) { } StorageRpc.BatchResponse response = storageRpc.batch(new StorageRpc.BatchRequest(toDelete, toUpdate, toGet)); - List> deletes = transformBatchResult( - toDelete, response.deletes, Functions.identity()); - List> updates = transformBatchResult( - toUpdate, response.updates, BlobInfo.FROM_PB_FUNCTION); - List> gets = transformBatchResult( - toGet, response.gets, BlobInfo.FROM_PB_FUNCTION); + List> deletes = + transformBatchResult(toDelete, response.deletes, DELETE_FUNCTION); + List> updates = + transformBatchResult(toUpdate, response.updates, Blob.BLOB_FROM_PB_FUNCTION); + List> gets = + transformBatchResult(toGet, response.gets, Blob.BLOB_FROM_PB_FUNCTION); return new BatchResponse(deletes, updates, gets); } private List> transformBatchResult( Iterable>> request, - Map> results, Function transform) { + Map> results, + Function, O> transform) { List> response = Lists.newArrayListWithCapacity(results.size()); for (Tuple tuple : request) { Tuple result = results.get(tuple.x()); @@ -489,7 +500,8 @@ private List> transformBatch response.add(new BatchResponse.Result(exception)); } else { response.add(object != null - ? BatchResponse.Result.of(transform.apply(object)) : BatchResponse.Result.empty()); + ? BatchResponse.Result.of(transform.apply(Tuple.of((Storage) this, object))) + : BatchResponse.Result.empty()); } } return response; @@ -587,7 +599,7 @@ public URL signUrl(BlobInfo blobInfo, long duration, TimeUnit unit, SignUrlOptio } @Override - public List get(BlobId... blobIds) { + public List get(BlobId... blobIds) { BatchRequest.Builder requestBuilder = BatchRequest.builder(); for (BlobId blob : blobIds) { requestBuilder.get(blob); @@ -597,7 +609,7 @@ public List get(BlobId... blobIds) { } @Override - public List update(BlobInfo... blobInfos) { + public List update(BlobInfo... blobInfos) { BatchRequest.Builder requestBuilder = BatchRequest.builder(); for (BlobInfo blobInfo : blobInfos) { requestBuilder.update(blobInfo); diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java index fda14ea2e808..181e63b08d0b 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/package-info.java @@ -17,23 +17,32 @@ /** * A client to Google Cloud Storage. * - *

    Here's a simple usage example for using gcloud-java from App/Compute Engine: + *

    Here's a simple usage example for using gcloud-java from App/Compute Engine. This example + * shows how to create a Storage blob. For the complete source code see + * + * CreateBlob.java. *

     {@code
      * Storage storage = StorageOptions.defaultInstance().service();
      * BlobId blobId = BlobId.of("bucket", "blob_name");
    - * Blob blob = Blob.get(storage, blobId);
    - * if (blob == null) {
    - *   BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    - *   storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    - * } else {
    - *   System.out.println("Updating content for " + blobId.name());
    + * BlobInfo blobInfo = BlobInfo.builder(blobId).contentType("text/plain").build();
    + * Blob blob = storage.create(blobInfo, "Hello, Cloud Storage!".getBytes(UTF_8));
    + * }
    + *

    + * This second example shows how to update the blob's content if the blob exists. For the complete + * source code see + * + * UpdateBlob.java. + *

     {@code
    + * Storage storage = StorageOptions.defaultInstance().service();
    + * BlobId blobId = BlobId.of("bucket", "blob_name");
    + * Blob blob = storage.get(blobId);
    + * if (blob != null) {
      *   byte[] prevContent = blob.content();
      *   System.out.println(new String(prevContent, UTF_8));
      *   WritableByteChannel channel = blob.writer();
      *   channel.write(ByteBuffer.wrap("Updated content".getBytes(UTF_8)));
      *   channel.close();
      * }}
    - * *

    When using gcloud-java from outside of App/Compute Engine, you have to specify a * project ID and diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/FakeStorageRpc.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/FakeStorageRpc.java new file mode 100644 index 000000000000..a04e8b73c1fd --- /dev/null +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/FakeStorageRpc.java @@ -0,0 +1,277 @@ +package com.google.gcloud.storage.testing; + +import com.google.api.services.storage.model.Bucket; +import com.google.api.services.storage.model.StorageObject; +import com.google.gcloud.spi.StorageRpc; +import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.StorageException; + +import java.io.IOException; +import java.io.InputStream; +import java.math.BigInteger; +import java.nio.file.FileAlreadyExistsException; +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; + +import javax.annotation.concurrent.NotThreadSafe; + +/** + * A bare-bones in-memory implementation of Storage, meant for testing. + * See LocalGcsHelper. + * + * This class is NOT thread-safe. + */ +@NotThreadSafe +public class FakeStorageRpc implements StorageRpc { + + // fullname -> metadata + Map stuff = new HashMap<>(); + // fullname -> contents + Map contents = new HashMap<>(); + // fullname -> future contents that will be visible on close. + Map futureContents = new HashMap<>(); + + private final boolean throwIfOption; + + /** + * @param throwIfOption if true, we throw when given any option. + */ + public FakeStorageRpc(boolean throwIfOption) { + this.throwIfOption = throwIfOption; + } + + // remove all files + void reset() { + stuff = new HashMap<>(); + contents = new HashMap<>(); + } + + @Override + public Bucket create(Bucket bucket, Map options) throws StorageException { + throw new UnsupportedOperationException(); + } + + @Override + public StorageObject create(StorageObject object, InputStream content, Map options) + throws StorageException { + potentiallyThrow(options); + String key = fullname(object); + stuff.put(key, object); + try { + contents.put(key, com.google.common.io.ByteStreams.toByteArray(content)); + } catch (IOException e) { + throw new StorageException(e); + } + // TODO: crc, etc + return object; + } + + @Override + public Tuple> list(Map options) throws StorageException { + throw new UnsupportedOperationException(); + } + + @Override + public Tuple> list(String bucket, Map options) + throws StorageException { + potentiallyThrow(options); + return null; + } + + /** + * Returns the requested bucket or {@code null} if not found. + */ + @Override + public Bucket get(Bucket bucket, Map options) throws StorageException { + potentiallyThrow(options); + return null; + } + + /** + * Returns the requested storage object or {@code null} if not found. + */ + @Override + public StorageObject get(StorageObject object, Map options) throws StorageException { + // we allow the "ID" option because we need to, but then we give a whole answer anyways + // because the caller won't mind the extra fields. + if (throwIfOption && !options.isEmpty() && options.size()>1 + && options.keySet().toArray()[0] != Storage.BlobGetOption.fields(Storage.BlobField.ID)) { + throw new UnsupportedOperationException(); + } + + String key = fullname(object); + if (stuff.containsKey(key)) { + StorageObject ret = stuff.get(key); + if (contents.containsKey(key)) { + ret.setSize(BigInteger.valueOf(contents.get(key).length)); + } + ret.setId(key); + return ret; + } + return null; + } + + @Override + public Bucket patch(Bucket bucket, Map options) throws StorageException { + potentiallyThrow(options); + return null; + } + + @Override + public StorageObject patch(StorageObject storageObject, Map options) + throws StorageException { + potentiallyThrow(options); + return null; + } + + @Override + public boolean delete(Bucket bucket, Map options) throws StorageException { + return false; + } + + @Override + public boolean delete(StorageObject object, Map options) throws StorageException { + String key = fullname(object); + contents.remove(key); + return null != stuff.remove(key); + } + + @Override + public BatchResponse batch(BatchRequest request) throws StorageException { + return null; + } + + @Override + public StorageObject compose(Iterable sources, StorageObject target, + Map targetOptions) throws StorageException { + return null; + } + + @Override + public byte[] load(StorageObject storageObject, Map options) throws StorageException { + String key = fullname(storageObject); + if (!contents.containsKey(key)) { + throw new StorageException(404, "File not found: " + key); + } + return contents.get(key); + } + + @Override + public Tuple read( + StorageObject from, Map options, long zposition, int zbytes) + throws StorageException { + potentiallyThrow(options); + String key = fullname(from); + if (!contents.containsKey(key)) { + throw new StorageException(404, "File not found: " + key); + } + long position = zposition; + int bytes = zbytes; + if (position < 0) { + position = 0; + } + byte[] full = contents.get(key); + if ((int) position + bytes > full.length) { + bytes = full.length - (int) position; + } + if (bytes <= 0) { + // special case: you're trying to read past the end + return Tuple.of("etag-goes-here", new byte[0]); + } + byte[] ret = new byte[bytes]; + System.arraycopy(full, (int) position, ret, 0, bytes); + return Tuple.of("etag-goes-here", ret); + } + + @Override + public String open(StorageObject object, Map options) throws StorageException { + String key = fullname(object); + boolean mustNotExist = false; + for (Option option : options.keySet()) { + if (option instanceof StorageRpc.Option) { + // this is a bit of a hack, since we don't implement generations. + if ((StorageRpc.Option) option == Option.IF_GENERATION_MATCH + && ((Long) options.get(option)).longValue() == 0L) { + mustNotExist = true; + } + } + } + if (mustNotExist && stuff.containsKey(key)) { + throw new StorageException(new FileAlreadyExistsException(key)); + } + stuff.put(key, object); + + return fullname(object); + } + + @Override + public void write(String uploadId, byte[] toWrite, int toWriteOffset, long destOffset, + int length, boolean last) throws StorageException { + // this may have a lot more allocations than ideal, but it'll work. + byte[] bytes; + if (futureContents.containsKey(uploadId)) { + bytes = futureContents.get(uploadId); + if (bytes.length < length + destOffset) { + bytes = new byte[(int) (length + destOffset)]; + } + } else { + bytes = new byte[(int) (length + destOffset)]; + } + System.arraycopy(toWrite, toWriteOffset, bytes, (int) destOffset, length); + // we want to mimic the GCS behavior that file contents are only visible on close. + if (last) { + contents.put(uploadId, bytes); + futureContents.remove(uploadId); + } else { + futureContents.put(uploadId, bytes); + } + } + + @Override + public RewriteResponse openRewrite(RewriteRequest rewriteRequest) throws StorageException { + String sourceKey = fullname(rewriteRequest.source); + // a little hackish, just good enough for the tests to work. + if (!contents.containsKey(sourceKey)) { + throw new StorageException(404, "File not found: " + sourceKey); + } + + boolean mustNotExist = false; + for (Option option : rewriteRequest.targetOptions.keySet()) { + if (option instanceof StorageRpc.Option) { + // this is a bit of a hack, since we don't implement generations. + if ((StorageRpc.Option) option == Option.IF_GENERATION_MATCH + && ((Long) rewriteRequest.targetOptions.get(option)).longValue() == 0L) { + mustNotExist = true; + } + } + } + + String destKey = fullname(rewriteRequest.target); + if (mustNotExist && contents.containsKey(destKey)) { + throw new StorageException(new FileAlreadyExistsException(destKey)); + } + + stuff.put(destKey, rewriteRequest.target); + + byte[] data = contents.get(sourceKey); + contents.put(destKey, Arrays.copyOf(data, data.length)); + return new RewriteResponse(rewriteRequest, rewriteRequest.target, data.length, true, + "rewriteToken goes here", data.length); + } + + @Override + public RewriteResponse continueRewrite(RewriteResponse previousResponse) throws StorageException { + throw new UnsupportedOperationException(); + } + + private String fullname(StorageObject so) { + return (so.getBucket() + "/" + so.getName()); + } + + private void potentiallyThrow(Map options) throws UnsupportedOperationException { + if (throwIfOption && !options.isEmpty()) { + throw new UnsupportedOperationException(); + } + } +} diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/LocalGcsHelper.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/LocalGcsHelper.java new file mode 100644 index 000000000000..1c3a64452441 --- /dev/null +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/LocalGcsHelper.java @@ -0,0 +1,68 @@ +/* + * Copyright 2015 Google Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.google.gcloud.storage.testing; + +import com.google.gcloud.spi.ServiceRpcFactory; +import com.google.gcloud.spi.StorageRpc; +import com.google.gcloud.storage.StorageOptions; + +/** + * Utility to create an in-memory storage configuration for testing. Storage options can be + * obtained via the {@link #options()} method. Returned options will point to FakeStorageRpc. + */ +public class LocalGcsHelper { + + // used for testing. Will throw if you pass it an option. + private static final FakeStorageRpc instance = new FakeStorageRpc(true); + + /** + * Returns a {@link StorageOptions} that use the static FakeStorageRpc instance, + * and resets it first so you start from a clean slate. + * That instance will throw if you pass it any option. * + */ + public static StorageOptions options() { + instance.reset(); + return StorageOptions.builder() + .projectId("dummy-project-for-testing") + .serviceRpcFactory( + new ServiceRpcFactory() { + @Override + public StorageRpc create(StorageOptions options) { + return instance; + } + }) + .build(); + } + + /** + * Returns a {@link StorageOptions} that creates a new FakeStorageRpc instance + * with the given option. + */ + public static StorageOptions customOptions(final boolean throwIfOptions) { + return StorageOptions.builder() + .projectId("dummy-project-for-testing") + .serviceRpcFactory( + new ServiceRpcFactory() { + @Override + public StorageRpc create(StorageOptions options) { + return new FakeStorageRpc(throwIfOptions); + } + }) + .build(); + } + +} diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java index 024aa04eba1b..1287ede746d5 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/RemoteGcsHelper.java @@ -20,6 +20,7 @@ import com.google.gcloud.RetryParams; import com.google.gcloud.storage.BlobInfo; import com.google.gcloud.storage.Storage; +import com.google.gcloud.storage.Storage.BlobListOption; import com.google.gcloud.storage.StorageException; import com.google.gcloud.storage.StorageOptions; @@ -173,8 +174,8 @@ public DeleteBucketTask(Storage storage, String bucket) { @Override public Boolean call() { while (true) { - for (BlobInfo info : storage.list(bucket).values()) { - storage.delete(bucket, info.name()); + for (BlobInfo info : storage.list(bucket, BlobListOption.versions(true)).values()) { + storage.delete(info.blobId()); } try { storage.delete(bucket); diff --git a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/package-info.java b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/package-info.java index 8afdd8a9660d..2ea1866f2966 100644 --- a/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/package-info.java +++ b/gcloud-java-storage/src/main/java/com/google/gcloud/storage/testing/package-info.java @@ -15,7 +15,9 @@ */ /** - * A testing helper for Google Cloud Storage. + * Two testing helpers for Google Cloud Storage. + * + * RemoteGcsHelper helps with testing on the actual cloud. * *

    A simple usage example: * @@ -32,6 +34,21 @@ * RemoteGcsHelper.forceDelete(storage, bucket, 5, TimeUnit.SECONDS); * }

    * + * LocalGcsHelper helps with testing on an in-memory filesystem (this is best for unit tests). + * Note that this filesystem isn't a complete implementation. In particular, it is not thread-safe. + * + *

    A simple usage example: + * + *

    + * CloudStorageFileSystemProvider.setGCloudOptions(LocalGcsHelper.options());
    + * Path path = Paths.get(URI.create("gs://bucket/wednesday"));
    + * thrown.expect(NoSuchFileException.class);
    + * Files.newByteChannel(path);
    + * 
    + * + *

    The first line ensures that the test uses the in-memory GCS instead of the real one + * (note that you have to set the cloud options before the first usage of the gs:// prefix). + * * @see * gcloud-java tools for testing */ diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java index 5985329e0183..eb45b8b51271 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BatchResponseTest.java @@ -22,22 +22,35 @@ import com.google.common.collect.ImmutableList; import com.google.gcloud.storage.BatchResponse.Result; +import org.easymock.EasyMock; +import org.junit.Before; import org.junit.Test; import java.util.List; public class BatchResponseTest { - private static final BlobInfo BLOB_INFO_1 = BlobInfo.builder("b", "o1").build(); - private static final BlobInfo BLOB_INFO_2 = BlobInfo.builder("b", "o2").build(); - private static final BlobInfo BLOB_INFO_3 = BlobInfo.builder("b", "o3").build(); + private Storage mockStorage; + private Blob blob1; + private Blob blob2; + private Blob blob3; + + @Before + public void setUp() { + mockStorage = EasyMock.createMock(Storage.class); + EasyMock.expect(mockStorage.options()).andReturn(null).times(3); + EasyMock.replay(mockStorage); + blob1 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o1").build())); + blob2 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o2").build())); + blob3 = new Blob(mockStorage, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "o3").build())); + } @Test public void testBatchResponse() { List> deletes = ImmutableList.of(Result.of(true), Result.of(false)); - List> updates = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); - List> gets = ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); + List> updates = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); + List> gets = ImmutableList.of(Result.of(blob2), Result.of(blob3)); BatchResponse response = new BatchResponse(deletes, updates, gets); assertEquals(deletes, response.deletes()); assertEquals(updates, response.updates()); @@ -47,14 +60,13 @@ public void testBatchResponse() { @Test public void testEquals() { List> deletes = ImmutableList.of(Result.of(true), Result.of(false)); - List> updates = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); - List> gets = ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); + List> updates = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); + List> gets = ImmutableList.of(Result.of(blob2), Result.of(blob3)); List> otherDeletes = ImmutableList.of(Result.of(false), Result.of(true)); - List> otherUpdates = - ImmutableList.of(Result.of(BLOB_INFO_2), Result.of(BLOB_INFO_3)); - List> otherGets = - ImmutableList.of(Result.of(BLOB_INFO_1), Result.of(BLOB_INFO_2)); + List> otherUpdates = ImmutableList.of(Result.of(blob2), Result.of(blob3)); + List> otherGets = + ImmutableList.of(Result.of(blob1), Result.of(blob2)); BatchResponse response = new BatchResponse(deletes, updates, gets); BatchResponse responseEquals = new BatchResponse(deletes, updates, gets); BatchResponse responseNotEquals1 = new BatchResponse(otherDeletes, updates, gets); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java index ffb37e8c5032..5dc947df51f8 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobReadChannelTest.java @@ -39,6 +39,7 @@ import java.io.IOException; import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; import java.util.Arrays; import java.util.Map; import java.util.Random; @@ -156,15 +157,15 @@ public void testClose() { } @Test - public void testReadClosed() { + public void testReadClosed() throws IOException { replay(storageRpcMock); reader = new BlobReadChannel(options, BLOB_ID, EMPTY_RPC_OPTIONS); reader.close(); try { ByteBuffer readBuffer = ByteBuffer.allocate(DEFAULT_CHUNK_SIZE); reader.read(readBuffer); - fail("Expected BlobReadChannel read to throw IOException"); - } catch (IOException ex) { + fail("Expected BlobReadChannel read to throw ClosedChannelException"); + } catch (ClosedChannelException ex) { // expected } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java index 586e7fd0fd39..c7508593f8c9 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BlobTest.java @@ -16,22 +16,28 @@ package com.google.gcloud.storage; +import static com.google.gcloud.storage.Acl.Project.ProjectRole.VIEWERS; +import static com.google.gcloud.storage.Acl.Role.READER; +import static com.google.gcloud.storage.Acl.Role.WRITER; import static org.easymock.EasyMock.capture; import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; +import static org.easymock.EasyMock.eq; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; -import com.google.api.client.util.Lists; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import com.google.gcloud.ReadChannel; +import com.google.gcloud.storage.Acl.Project; +import com.google.gcloud.storage.Acl.User; import com.google.gcloud.storage.Storage.CopyRequest; import org.easymock.Capture; @@ -40,25 +46,65 @@ import org.junit.Test; import java.net.URL; -import java.util.Arrays; import java.util.List; +import java.util.Map; import java.util.concurrent.TimeUnit; public class BlobTest { + private static final List ACL = ImmutableList.of( + Acl.of(User.ofAllAuthenticatedUsers(), READER), Acl.of(new Project(VIEWERS, "p1"), WRITER)); + private static final Integer COMPONENT_COUNT = 2; + private static final String CONTENT_TYPE = "text/html"; + private static final String CACHE_CONTROL = "cache"; + private static final String CONTENT_DISPOSITION = "content-disposition"; + private static final String CONTENT_ENCODING = "UTF-8"; + private static final String CONTENT_LANGUAGE = "En"; + private static final String CRC32 = "0xFF00"; + private static final Long DELETE_TIME = System.currentTimeMillis(); + private static final String ETAG = "0xFF00"; + private static final Long GENERATION = 1L; + private static final String ID = "B/N:1"; + private static final String MD5 = "0xFF00"; + private static final String MEDIA_LINK = "http://media/b/n"; + private static final Map METADATA = ImmutableMap.of("n1", "v1", "n2", "v2"); + private static final Long META_GENERATION = 10L; + private static final User OWNER = new User("user@gmail.com"); + private static final String SELF_LINK = "http://storage/b/n"; + private static final Long SIZE = 1024L; + private static final Long UPDATE_TIME = DELETE_TIME - 1L; + private static final BlobInfo FULL_BLOB_INFO = BlobInfo.builder("b", "n", GENERATION) + .acl(ACL) + .componentCount(COMPONENT_COUNT) + .contentType(CONTENT_TYPE) + .cacheControl(CACHE_CONTROL) + .contentDisposition(CONTENT_DISPOSITION) + .contentEncoding(CONTENT_ENCODING) + .contentLanguage(CONTENT_LANGUAGE) + .crc32c(CRC32) + .deleteTime(DELETE_TIME) + .etag(ETAG) + .id(ID) + .md5(MD5) + .mediaLink(MEDIA_LINK) + .metadata(METADATA) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .size(SIZE) + .updateTime(UPDATE_TIME) + .build(); private static final BlobInfo BLOB_INFO = BlobInfo.builder("b", "n").metageneration(42L).build(); - private static final BlobId[] BLOB_ID_ARRAY = {BlobId.of("b1", "n1"), - BlobId.of("b2", "n2"), BlobId.of("b3", "n3")}; - private static final BlobInfo[] BLOB_INFO_ARRAY = {BlobInfo.builder("b1", "n1").build(), - BlobInfo.builder("b2", "n2").build(), BlobInfo.builder("b3", "n3").build()}; private Storage storage; private Blob blob; + private Blob expectedBlob; + private Storage serviceMockReturnsOptions = createMock(Storage.class); + private StorageOptions mockOptions = createMock(StorageOptions.class); @Before - public void setUp() throws Exception { + public void setUp() { storage = createStrictMock(Storage.class); - blob = new Blob(storage, BLOB_INFO); } @After @@ -66,91 +112,122 @@ public void tearDown() throws Exception { verify(storage); } - @Test - public void testInfo() throws Exception { - assertEquals(BLOB_INFO, blob.info()); - replay(storage); + private void initializeExpectedBlob(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); + expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(BLOB_INFO)); + } + + private void initializeBlob() { + blob = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO)); } @Test public void testExists_True() throws Exception { + initializeExpectedBlob(1); Storage.BlobGetOption[] expectedOptions = {Storage.BlobGetOption.fields()}; - expect(storage.get(BLOB_INFO.blobId(), expectedOptions)).andReturn(BLOB_INFO); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(expectedBlob.blobId(), expectedOptions)).andReturn(expectedBlob); replay(storage); + initializeBlob(); assertTrue(blob.exists()); } @Test public void testExists_False() throws Exception { Storage.BlobGetOption[] expectedOptions = {Storage.BlobGetOption.fields()}; + expect(storage.options()).andReturn(null); expect(storage.get(BLOB_INFO.blobId(), expectedOptions)).andReturn(null); replay(storage); + initializeBlob(); assertFalse(blob.exists()); } @Test public void testContent() throws Exception { + initializeExpectedBlob(2); byte[] content = {1, 2}; + expect(storage.options()).andReturn(mockOptions); expect(storage.readAllBytes(BLOB_INFO.blobId())).andReturn(content); replay(storage); + initializeBlob(); assertArrayEquals(content, blob.content()); } @Test public void testReload() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(updatedInfo); + initializeExpectedBlob(2); + Blob expectedReloadedBlob = expectedBlob.toBuilder().cacheControl("c").build(); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])) + .andReturn(expectedReloadedBlob); replay(storage); + initializeBlob(); Blob updatedBlob = blob.reload(); - assertSame(storage, updatedBlob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + assertEquals(expectedReloadedBlob, updatedBlob); } @Test public void testReloadNull() throws Exception { + initializeExpectedBlob(1); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); replay(storage); - assertNull(blob.reload()); + initializeBlob(); + Blob reloadedBlob = blob.reload(); + assertNull(reloadedBlob); } @Test public void testReloadWithOptions() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); + initializeExpectedBlob(2); + Blob expectedReloadedBlob = expectedBlob.toBuilder().cacheControl("c").build(); Storage.BlobGetOption[] options = {Storage.BlobGetOption.metagenerationMatch(42L)}; - expect(storage.get(BLOB_INFO.blobId(), options)).andReturn(updatedInfo); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BLOB_INFO.blobId(), options)).andReturn(expectedReloadedBlob); replay(storage); + initializeBlob(); Blob updatedBlob = blob.reload(Blob.BlobSourceOption.metagenerationMatch()); - assertSame(storage, updatedBlob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + assertEquals(expectedReloadedBlob, updatedBlob); } @Test public void testUpdate() throws Exception { - BlobInfo updatedInfo = BLOB_INFO.toBuilder().cacheControl("c").build(); - expect(storage.update(updatedInfo, new Storage.BlobTargetOption[0])).andReturn(updatedInfo); + initializeExpectedBlob(2); + Blob expectedUpdatedBlob = expectedBlob.toBuilder().cacheControl("c").build(); + expect(storage.options()).andReturn(mockOptions).times(2); + expect(storage.update(eq(expectedUpdatedBlob), new Storage.BlobTargetOption[0])) + .andReturn(expectedUpdatedBlob); replay(storage); - Blob updatedBlob = blob.update(updatedInfo); - assertSame(storage, blob.storage()); - assertEquals(updatedInfo, updatedBlob.info()); + initializeBlob(); + Blob updatedBlob = new Blob(storage, new BlobInfo.BuilderImpl(expectedUpdatedBlob)); + Blob actualUpdatedBlob = updatedBlob.update(); + assertEquals(expectedUpdatedBlob, actualUpdatedBlob); } @Test public void testDelete() throws Exception { + initializeExpectedBlob(2); + expect(storage.options()).andReturn(mockOptions); expect(storage.delete(BLOB_INFO.blobId(), new Storage.BlobSourceOption[0])).andReturn(true); replay(storage); + initializeBlob(); assertTrue(blob.delete()); } @Test public void testCopyToBucket() throws Exception { + initializeExpectedBlob(2); BlobInfo target = BlobInfo.builder(BlobId.of("bt", "n")).build(); CopyWriter copyWriter = createMock(CopyWriter.class); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo("bt"); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -158,14 +235,17 @@ public void testCopyToBucket() throws Exception { @Test public void testCopyTo() throws Exception { + initializeExpectedBlob(2); BlobInfo target = BlobInfo.builder(BlobId.of("bt", "nt")).build(); CopyWriter copyWriter = createMock(CopyWriter.class); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo("bt", "nt"); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -173,15 +253,18 @@ public void testCopyTo() throws Exception { @Test public void testCopyToBlobId() throws Exception { + initializeExpectedBlob(2); BlobId targetId = BlobId.of("bt", "nt"); CopyWriter copyWriter = createMock(CopyWriter.class); BlobInfo target = BlobInfo.builder(targetId).build(); Capture capturedCopyRequest = Capture.newInstance(); + expect(storage.options()).andReturn(mockOptions); expect(storage.copy(capture(capturedCopyRequest))).andReturn(copyWriter); replay(storage); + initializeBlob(); CopyWriter returnedCopyWriter = blob.copyTo(targetId); assertEquals(copyWriter, returnedCopyWriter); - assertEquals(capturedCopyRequest.getValue().source(), blob.id()); + assertEquals(capturedCopyRequest.getValue().source(), blob.blobId()); assertEquals(capturedCopyRequest.getValue().target(), target); assertTrue(capturedCopyRequest.getValue().sourceOptions().isEmpty()); assertTrue(capturedCopyRequest.getValue().targetOptions().isEmpty()); @@ -189,174 +272,93 @@ public void testCopyToBlobId() throws Exception { @Test public void testReader() throws Exception { + initializeExpectedBlob(2); ReadChannel channel = createMock(ReadChannel.class); + expect(storage.options()).andReturn(mockOptions); expect(storage.reader(BLOB_INFO.blobId())).andReturn(channel); replay(storage); + initializeBlob(); assertSame(channel, blob.reader()); } @Test public void testWriter() throws Exception { + initializeExpectedBlob(2); BlobWriteChannel channel = createMock(BlobWriteChannel.class); - expect(storage.writer(BLOB_INFO)).andReturn(channel); + expect(storage.options()).andReturn(mockOptions); + expect(storage.writer(eq(expectedBlob))).andReturn(channel); replay(storage); + initializeBlob(); assertSame(channel, blob.writer()); } @Test public void testSignUrl() throws Exception { + initializeExpectedBlob(2); URL url = new URL("http://localhost:123/bla"); - expect(storage.signUrl(BLOB_INFO, 100, TimeUnit.SECONDS)).andReturn(url); + expect(storage.options()).andReturn(mockOptions); + expect(storage.signUrl(expectedBlob, 100, TimeUnit.SECONDS)).andReturn(url); replay(storage); + initializeBlob(); assertEquals(url, blob.signUrl(100, TimeUnit.SECONDS)); } @Test - public void testGetSome() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, BLOB_ID_ARRAY[0], BLOB_ID_ARRAY[1], BLOB_ID_ARRAY[2]); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testGetSomeList() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, Arrays.asList(BLOB_ID_ARRAY)); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testGetSomeNull() throws Exception { - List blobInfoList = Arrays.asList(BLOB_INFO_ARRAY[0], null, BLOB_INFO_ARRAY[2]); - expect(storage.get(BLOB_ID_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.get(storage, BLOB_ID_ARRAY[0], BLOB_ID_ARRAY[1], BLOB_ID_ARRAY[2]); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - if (blobInfoList.get(i) != null) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } else { - assertNull(result.get(i)); - } - } - } - - @Test - public void testUpdateNone() throws Exception { - replay(storage); - assertTrue(Blob.update(storage).isEmpty()); - } - - @Test - public void testUpdateSome() throws Exception { - List blobInfoList = Lists.newArrayListWithCapacity(BLOB_ID_ARRAY.length); - for (BlobInfo info : BLOB_INFO_ARRAY) { - blobInfoList.add(info.toBuilder().contentType("content").build()); - } - expect(storage.update(BLOB_INFO_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.update(storage, BLOB_INFO_ARRAY); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } - } - - @Test - public void testUpdateSomeNull() throws Exception { - List blobInfoList = Arrays.asList( - BLOB_INFO_ARRAY[0].toBuilder().contentType("content").build(), null, - BLOB_INFO_ARRAY[2].toBuilder().contentType("content").build()); - expect(storage.update(BLOB_INFO_ARRAY)).andReturn(blobInfoList); - replay(storage); - List result = Blob.update(storage, BLOB_INFO_ARRAY); - assertEquals(blobInfoList.size(), result.size()); - for (int i = 0; i < blobInfoList.size(); i++) { - if (blobInfoList.get(i) != null) { - assertEquals(blobInfoList.get(i), result.get(i).info()); - } else { - assertNull(result.get(i)); - } - } - } - - @Test - public void testDeleteNone() throws Exception { - replay(storage); - assertTrue(Blob.delete(storage).isEmpty()); - } - - @Test - public void testDeleteSome() throws Exception { - List deleteResult = Arrays.asList(true, true, true); - expect(storage.delete(BLOB_ID_ARRAY)).andReturn(deleteResult); - replay(storage); - List result = Blob.delete(storage, BLOB_ID_ARRAY); - assertEquals(deleteResult.size(), result.size()); - for (int i = 0; i < deleteResult.size(); i++) { - assertEquals(deleteResult.get(i), result.get(i)); - } - } - - @Test - public void testGetFromString() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(BLOB_INFO); - replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name()); - assertEquals(BLOB_INFO, loadedBlob.info()); - } - - @Test - public void testGetFromId() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(BLOB_INFO); - replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.blobId()); - assertNotNull(loadedBlob); - assertEquals(BLOB_INFO, loadedBlob.info()); - } - - @Test - public void testGetFromStringNull() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); - replay(storage); - assertNull(Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name())); - } - - @Test - public void testGetFromIdNull() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), new Storage.BlobGetOption[0])).andReturn(null); - replay(storage); - assertNull(Blob.get(storage, BLOB_INFO.blobId())); - } - - @Test - public void testGetFromStringWithOptions() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L))) - .andReturn(BLOB_INFO); + public void testToBuilder() { + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - Blob loadedBlob = Blob.get(storage, BLOB_INFO.bucket(), BLOB_INFO.name(), - Storage.BlobGetOption.generationMatch(42L)); - assertEquals(BLOB_INFO, loadedBlob.info()); + Blob fullBlob = new Blob(storage, new BlobInfo.BuilderImpl(FULL_BLOB_INFO)); + assertEquals(fullBlob, fullBlob.toBuilder().build()); + Blob simpleBlob = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO)); + assertEquals(simpleBlob, simpleBlob.toBuilder().build()); } @Test - public void testGetFromIdWithOptions() throws Exception { - expect(storage.get(BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L))) - .andReturn(BLOB_INFO); + public void testBuilder() { + initializeExpectedBlob(4); + expect(storage.options()).andReturn(mockOptions).times(2); replay(storage); - Blob loadedBlob = - Blob.get(storage, BLOB_INFO.blobId(), Storage.BlobGetOption.generationMatch(42L)); - assertNotNull(loadedBlob); - assertEquals(BLOB_INFO, loadedBlob.info()); + Blob.Builder builder = new Blob.Builder(new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO))); + Blob blob = builder.acl(ACL) + .componentCount(COMPONENT_COUNT) + .contentType(CONTENT_TYPE) + .cacheControl(CACHE_CONTROL) + .contentDisposition(CONTENT_DISPOSITION) + .contentEncoding(CONTENT_ENCODING) + .contentLanguage(CONTENT_LANGUAGE) + .crc32c(CRC32) + .deleteTime(DELETE_TIME) + .etag(ETAG) + .id(ID) + .md5(MD5) + .mediaLink(MEDIA_LINK) + .metadata(METADATA) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .size(SIZE) + .updateTime(UPDATE_TIME) + .build(); + assertEquals("b", blob.bucket()); + assertEquals("n", blob.name()); + assertEquals(ACL, blob.acl()); + assertEquals(COMPONENT_COUNT, blob.componentCount()); + assertEquals(CONTENT_TYPE, blob.contentType()); + assertEquals(CACHE_CONTROL, blob.cacheControl()); + assertEquals(CONTENT_DISPOSITION, blob.contentDisposition()); + assertEquals(CONTENT_ENCODING, blob.contentEncoding()); + assertEquals(CONTENT_LANGUAGE, blob.contentLanguage()); + assertEquals(CRC32, blob.crc32c()); + assertEquals(DELETE_TIME, blob.deleteTime()); + assertEquals(ETAG, blob.etag()); + assertEquals(ID, blob.id()); + assertEquals(MD5, blob.md5()); + assertEquals(MEDIA_LINK, blob.mediaLink()); + assertEquals(METADATA, blob.metadata()); + assertEquals(META_GENERATION, blob.metageneration()); + assertEquals(OWNER, blob.owner()); + assertEquals(SELF_LINK, blob.selfLink()); + assertEquals(SIZE, blob.size()); + assertEquals(UPDATE_TIME, blob.updateTime()); } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java index 4e253033c6f2..236411e0c2d8 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/BucketTest.java @@ -16,27 +16,35 @@ package com.google.gcloud.storage; +import static com.google.gcloud.storage.Acl.Project.ProjectRole.VIEWERS; +import static com.google.gcloud.storage.Acl.Role.READER; +import static com.google.gcloud.storage.Acl.Role.WRITER; import static org.easymock.EasyMock.capture; +import static org.easymock.EasyMock.createMock; import static org.easymock.EasyMock.createStrictMock; import static org.easymock.EasyMock.expect; import static org.easymock.EasyMock.replay; import static org.easymock.EasyMock.verify; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; import com.google.gcloud.PageImpl; +import com.google.gcloud.storage.Acl.Project; +import com.google.gcloud.storage.Acl.User; import com.google.gcloud.storage.BatchResponse.Result; +import com.google.gcloud.storage.BucketInfo.AgeDeleteRule; +import com.google.gcloud.storage.BucketInfo.DeleteRule; import org.easymock.Capture; import org.junit.After; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.ExpectedException; import java.io.ByteArrayInputStream; import java.io.InputStream; @@ -48,20 +56,57 @@ public class BucketTest { + private static final List ACL = ImmutableList.of( + Acl.of(User.ofAllAuthenticatedUsers(), READER), Acl.of(new Project(VIEWERS, "p1"), WRITER)); + private static final String ETAG = "0xFF00"; + private static final String ID = "B/N:1"; + private static final Long META_GENERATION = 10L; + private static final User OWNER = new User("user@gmail.com"); + private static final String SELF_LINK = "http://storage/b/n"; + private static final Long CREATE_TIME = System.currentTimeMillis(); + private static final List CORS = Collections.singletonList(Cors.builder().build()); + private static final List DEFAULT_ACL = + Collections.singletonList(Acl.of(User.ofAllAuthenticatedUsers(), WRITER)); + private static final List DELETE_RULES = + Collections.singletonList(new AgeDeleteRule(5)); + private static final String INDEX_PAGE = "index.html"; + private static final String NOT_FOUND_PAGE = "error.html"; + private static final String LOCATION = "ASIA"; + private static final String STORAGE_CLASS = "STANDARD"; + private static final Boolean VERSIONING_ENABLED = true; + private static final BucketInfo FULL_BUCKET_INFO = BucketInfo.builder("b") + .acl(ACL) + .etag(ETAG) + .id(ID) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .cors(CORS) + .createTime(CREATE_TIME) + .defaultAcl(DEFAULT_ACL) + .deleteRules(DELETE_RULES) + .indexPage(INDEX_PAGE) + .notFoundPage(NOT_FOUND_PAGE) + .location(LOCATION) + .storageClass(STORAGE_CLASS) + .versioningEnabled(VERSIONING_ENABLED) + .build(); private static final BucketInfo BUCKET_INFO = BucketInfo.builder("b").metageneration(42L).build(); - private static final Iterable BLOB_INFO_RESULTS = ImmutableList.of( - BlobInfo.builder("b", "n1").build(), - BlobInfo.builder("b", "n2").build(), - BlobInfo.builder("b", "n3").build()); private static final String CONTENT_TYPE = "text/plain"; private Storage storage; + private Storage serviceMockReturnsOptions = createMock(Storage.class); + private StorageOptions mockOptions = createMock(StorageOptions.class); private Bucket bucket; + private Bucket expectedBucket; + private Iterable blobResults; + + @Rule + public ExpectedException thrown = ExpectedException.none(); @Before - public void setUp() throws Exception { + public void setUp() { storage = createStrictMock(Storage.class); - bucket = new Bucket(storage, BUCKET_INFO); } @After @@ -69,124 +114,165 @@ public void tearDown() throws Exception { verify(storage); } - @Test - public void testInfo() throws Exception { - assertEquals(BUCKET_INFO, bucket.info()); - replay(storage); + private void initializeExpectedBucket(int optionsCalls) { + expect(serviceMockReturnsOptions.options()).andReturn(mockOptions).times(optionsCalls); + replay(serviceMockReturnsOptions); + expectedBucket = new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(BUCKET_INFO)); + blobResults = ImmutableList.of( + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n1").build())), + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n2").build())), + new Blob(serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n3").build()))); + } + + private void initializeBucket() { + bucket = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO)); } @Test public void testExists_True() throws Exception { + initializeExpectedBucket(4); Storage.BucketGetOption[] expectedOptions = {Storage.BucketGetOption.fields()}; - expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(BUCKET_INFO); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(expectedBucket); replay(storage); + initializeBucket(); assertTrue(bucket.exists()); } @Test public void testExists_False() throws Exception { + initializeExpectedBucket(4); Storage.BucketGetOption[] expectedOptions = {Storage.BucketGetOption.fields()}; + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BUCKET_INFO.name(), expectedOptions)).andReturn(null); replay(storage); + initializeBucket(); assertFalse(bucket.exists()); } @Test public void testReload() throws Exception { + initializeExpectedBucket(5); BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); - expect(storage.get(updatedInfo.name())).andReturn(updatedInfo); + Bucket expectedUpdatedBucket = + new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(updatedInfo)); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(updatedInfo.name())).andReturn(expectedUpdatedBucket); replay(storage); + initializeBucket(); Bucket updatedBucket = bucket.reload(); - assertSame(storage, updatedBucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + assertEquals(expectedUpdatedBucket, updatedBucket); } @Test public void testReloadNull() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(BUCKET_INFO.name())).andReturn(null); replay(storage); + initializeBucket(); assertNull(bucket.reload()); } @Test public void testReloadWithOptions() throws Exception { + initializeExpectedBucket(5); BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); + Bucket expectedUpdatedBucket = + new Bucket(serviceMockReturnsOptions, new BucketInfo.BuilderImpl(updatedInfo)); + expect(storage.options()).andReturn(mockOptions); expect(storage.get(updatedInfo.name(), Storage.BucketGetOption.metagenerationMatch(42L))) - .andReturn(updatedInfo); + .andReturn(expectedUpdatedBucket); replay(storage); + initializeBucket(); Bucket updatedBucket = bucket.reload(Bucket.BucketSourceOption.metagenerationMatch()); - assertSame(storage, updatedBucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + assertEquals(expectedUpdatedBucket, updatedBucket); } @Test public void testUpdate() throws Exception { - BucketInfo updatedInfo = BUCKET_INFO.toBuilder().notFoundPage("p").build(); - expect(storage.update(updatedInfo)).andReturn(updatedInfo); + initializeExpectedBucket(5); + Bucket expectedUpdatedBucket = expectedBucket.toBuilder().notFoundPage("p").build(); + expect(storage.options()).andReturn(mockOptions).times(2); + expect(storage.update(expectedUpdatedBucket)).andReturn(expectedUpdatedBucket); replay(storage); - Bucket updatedBucket = bucket.update(updatedInfo); - assertSame(storage, bucket.storage()); - assertEquals(updatedInfo, updatedBucket.info()); + initializeBucket(); + Bucket updatedBucket = new Bucket(storage, new BucketInfo.BuilderImpl(expectedUpdatedBucket)); + Bucket actualUpdatedBucket = updatedBucket.update(); + assertEquals(expectedUpdatedBucket, actualUpdatedBucket); } @Test public void testDelete() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); expect(storage.delete(BUCKET_INFO.name())).andReturn(true); replay(storage); + initializeBucket(); assertTrue(bucket.delete()); } @Test public void testList() throws Exception { - StorageOptions storageOptions = createStrictMock(StorageOptions.class); - PageImpl blobInfoPage = new PageImpl<>(null, "c", BLOB_INFO_RESULTS); - expect(storage.list(BUCKET_INFO.name())).andReturn(blobInfoPage); - expect(storage.options()).andReturn(storageOptions); - expect(storageOptions.service()).andReturn(storage); - replay(storage, storageOptions); + initializeExpectedBucket(4); + PageImpl expectedBlobPage = new PageImpl<>(null, "c", blobResults); + expect(storage.options()).andReturn(mockOptions); + expect(storage.list(BUCKET_INFO.name())).andReturn(expectedBlobPage); + replay(storage); + initializeBucket(); Page blobPage = bucket.list(); - Iterator blobInfoIterator = blobInfoPage.values().iterator(); + Iterator blobInfoIterator = blobPage.values().iterator(); Iterator blobIterator = blobPage.values().iterator(); while (blobInfoIterator.hasNext() && blobIterator.hasNext()) { - assertEquals(blobInfoIterator.next(), blobIterator.next().info()); + assertEquals(blobInfoIterator.next(), blobIterator.next()); } assertFalse(blobInfoIterator.hasNext()); assertFalse(blobIterator.hasNext()); - assertEquals(blobInfoPage.nextPageCursor(), blobPage.nextPageCursor()); - verify(storageOptions); + assertEquals(expectedBlobPage.nextPageCursor(), blobPage.nextPageCursor()); } @Test public void testGet() throws Exception { - BlobInfo info = BlobInfo.builder("b", "n").build(); - expect(storage.get(BlobId.of(bucket.info().name(), "n"), new Storage.BlobGetOption[0])) - .andReturn(info); + initializeExpectedBucket(5); + Blob expectedBlob = new Blob( + serviceMockReturnsOptions, new BlobInfo.BuilderImpl(BlobInfo.builder("b", "n").build())); + expect(storage.options()).andReturn(mockOptions); + expect(storage.get(BlobId.of(expectedBucket.name(), "n"), new Storage.BlobGetOption[0])) + .andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.get("n"); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testGetAll() throws Exception { + initializeExpectedBucket(4); Capture capturedBatchRequest = Capture.newInstance(); - List> batchResultList = new LinkedList<>(); - for (BlobInfo info : BLOB_INFO_RESULTS) { + List> batchResultList = new LinkedList<>(); + for (Blob info : blobResults) { batchResultList.add(new Result<>(info)); } BatchResponse response = new BatchResponse(Collections.>emptyList(), - Collections.>emptyList(), batchResultList); + Collections.>emptyList(), batchResultList); + expect(storage.options()).andReturn(mockOptions); expect(storage.submit(capture(capturedBatchRequest))).andReturn(response); + expect(storage.options()).andReturn(mockOptions).times(3); replay(storage); + initializeBucket(); List blobs = bucket.get("n1", "n2", "n3"); Set blobInfoSet = capturedBatchRequest.getValue().toGet().keySet(); assertEquals(batchResultList.size(), blobInfoSet.size()); - for (BlobInfo info : BLOB_INFO_RESULTS) { + for (BlobInfo info : blobResults) { assertTrue(blobInfoSet.contains(info.blobId())); } Iterator blobIterator = blobs.iterator(); - Iterator> batchResultIterator = response.gets().iterator(); + Iterator> batchResultIterator = response.gets().iterator(); while (batchResultIterator.hasNext() && blobIterator.hasNext()) { - assertEquals(batchResultIterator.next().get(), blobIterator.next().info()); + assertEquals(batchResultIterator.next().get(), blobIterator.next()); } assertFalse(batchResultIterator.hasNext()); assertFalse(blobIterator.hasNext()); @@ -194,70 +280,251 @@ public void testGetAll() throws Exception { @Test public void testCreate() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; - expect(storage.create(info, content)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", content, CONTENT_TYPE); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testCreateNullContentType() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(Storage.DEFAULT_CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; - expect(storage.create(info, content)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", content, null); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateWithOptions() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 42L)) + .contentType(CONTENT_TYPE) + .metageneration(24L) + .build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + Storage.PredefinedAcl acl = Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS; + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content, Storage.BlobTargetOption.generationMatch(), + Storage.BlobTargetOption.metagenerationMatch(), + Storage.BlobTargetOption.predefinedAcl(acl))).andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = bucket.create("n", content, CONTENT_TYPE, + Bucket.BlobTargetOption.generationMatch(42L), + Bucket.BlobTargetOption.metagenerationMatch(24L), + Bucket.BlobTargetOption.predefinedAcl(acl)); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateNotExists() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 0L)).contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, content, Storage.BlobTargetOption.generationMatch())) + .andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.doesNotExist()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateWithWrongGenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.generationMatch(42L), + Bucket.BlobTargetOption.generationNotMatch(24L)); + } + + @Test + public void testCreateWithWrongMetagenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + bucket.create("n", content, CONTENT_TYPE, Bucket.BlobTargetOption.metagenerationMatch(42L), + Bucket.BlobTargetOption.metagenerationNotMatch(24L)); } @Test public void testCreateFromStream() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; InputStream streamContent = new ByteArrayInputStream(content); - expect(storage.create(info, streamContent)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", streamContent, CONTENT_TYPE); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test public void testCreateFromStreamNullContentType() throws Exception { + initializeExpectedBucket(5); BlobInfo info = BlobInfo.builder("b", "n").contentType(Storage.DEFAULT_CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); byte[] content = {0xD, 0xE, 0xA, 0xD}; InputStream streamContent = new ByteArrayInputStream(content); - expect(storage.create(info, streamContent)).andReturn(info); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent)).andReturn(expectedBlob); replay(storage); + initializeBucket(); Blob blob = bucket.create("n", streamContent, null); - assertEquals(info, blob.info()); + assertEquals(expectedBlob, blob); } @Test - public void testStaticGet() throws Exception { - expect(storage.get(BUCKET_INFO.name())).andReturn(BUCKET_INFO); + public void testCreateFromStreamWithOptions() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 42L)) + .contentType(CONTENT_TYPE) + .metageneration(24L) + .crc32c("crc") + .md5("md5") + .build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + Storage.PredefinedAcl acl = Storage.PredefinedAcl.ALL_AUTHENTICATED_USERS; + InputStream streamContent = new ByteArrayInputStream(content); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent, Storage.BlobWriteOption.generationMatch(), + Storage.BlobWriteOption.metagenerationMatch(), Storage.BlobWriteOption.predefinedAcl(acl), + Storage.BlobWriteOption.crc32cMatch(), Storage.BlobWriteOption.md5Match())) + .andReturn(expectedBlob); replay(storage); - Bucket loadedBucket = Bucket.get(storage, BUCKET_INFO.name()); - assertNotNull(loadedBucket); - assertEquals(BUCKET_INFO, loadedBucket.info()); + initializeBucket(); + Blob blob = bucket.create("n", streamContent, CONTENT_TYPE, + Bucket.BlobWriteOption.generationMatch(42L), + Bucket.BlobWriteOption.metagenerationMatch(24L), Bucket.BlobWriteOption.predefinedAcl(acl), + Bucket.BlobWriteOption.crc32cMatch("crc"), Bucket.BlobWriteOption.md5Match("md5")); + assertEquals(expectedBlob, blob); } @Test - public void testStaticGetNull() throws Exception { - expect(storage.get(BUCKET_INFO.name())).andReturn(null); + public void testCreateFromStreamNotExists() throws Exception { + initializeExpectedBucket(5); + BlobInfo info = BlobInfo.builder(BlobId.of("b", "n", 0L)).contentType(CONTENT_TYPE).build(); + Blob expectedBlob = new Blob(serviceMockReturnsOptions, new BlobInfo.BuilderImpl(info)); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + expect(storage.options()).andReturn(mockOptions); + expect(storage.create(info, streamContent, Storage.BlobWriteOption.generationMatch())) + .andReturn(expectedBlob); + replay(storage); + initializeBucket(); + Blob blob = + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.doesNotExist()); + assertEquals(expectedBlob, blob); + } + + @Test + public void testCreateFromStreamWithWrongGenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "Only one option of generationMatch, doesNotExist or generationNotMatch can be provided"); + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.generationMatch(42L), + Bucket.BlobWriteOption.generationNotMatch(24L)); + } + + @Test + public void testCreateFromStreamWithWrongMetagenerationOptions() throws Exception { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions); + replay(storage); + initializeBucket(); + byte[] content = {0xD, 0xE, 0xA, 0xD}; + InputStream streamContent = new ByteArrayInputStream(content); + thrown.expect(IllegalArgumentException.class); + thrown.expectMessage( + "metagenerationMatch and metagenerationNotMatch options can not be both provided"); + bucket.create("n", streamContent, CONTENT_TYPE, Bucket.BlobWriteOption.metagenerationMatch(42L), + Bucket.BlobWriteOption.metagenerationNotMatch(24L)); + } + + @Test + public void testToBuilder() { + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - assertNull(Bucket.get(storage, BUCKET_INFO.name())); + Bucket fullBucket = new Bucket(storage, new BucketInfo.BuilderImpl(FULL_BUCKET_INFO)); + assertEquals(fullBucket, fullBucket.toBuilder().build()); + Bucket simpleBlob = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO)); + assertEquals(simpleBlob, simpleBlob.toBuilder().build()); } @Test - public void testStaticGetWithOptions() throws Exception { - expect(storage.get(BUCKET_INFO.name(), Storage.BucketGetOption.fields())) - .andReturn(BUCKET_INFO); + public void testBuilder() { + initializeExpectedBucket(4); + expect(storage.options()).andReturn(mockOptions).times(4); replay(storage); - Bucket loadedBucket = - Bucket.get(storage, BUCKET_INFO.name(), Storage.BucketGetOption.fields()); - assertNotNull(loadedBucket); - assertEquals(BUCKET_INFO, loadedBucket.info()); + Bucket.Builder builder = + new Bucket.Builder(new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO))); + Bucket bucket = builder.acl(ACL) + .etag(ETAG) + .id(ID) + .metageneration(META_GENERATION) + .owner(OWNER) + .selfLink(SELF_LINK) + .cors(CORS) + .createTime(CREATE_TIME) + .defaultAcl(DEFAULT_ACL) + .deleteRules(DELETE_RULES) + .indexPage(INDEX_PAGE) + .notFoundPage(NOT_FOUND_PAGE) + .location(LOCATION) + .storageClass(STORAGE_CLASS) + .versioningEnabled(VERSIONING_ENABLED) + .build(); + assertEquals("b", bucket.name()); + assertEquals(ACL, bucket.acl()); + assertEquals(ETAG, bucket.etag()); + assertEquals(ID, bucket.id()); + assertEquals(META_GENERATION, bucket.metageneration()); + assertEquals(OWNER, bucket.owner()); + assertEquals(SELF_LINK, bucket.selfLink()); + assertEquals(CREATE_TIME, bucket.createTime()); + assertEquals(CORS, bucket.cors()); + assertEquals(DEFAULT_ACL, bucket.defaultAcl()); + assertEquals(DELETE_RULES, bucket.deleteRules()); + assertEquals(INDEX_PAGE, bucket.indexPage()); + assertEquals(NOT_FOUND_PAGE, bucket.notFoundPage()); + assertEquals(LOCATION, bucket.location()); + assertEquals(STORAGE_CLASS, bucket.storageClass()); + assertEquals(VERSIONING_ENABLED, bucket.versioningEnabled()); + assertEquals(storage.options(), bucket.storage().options()); } } diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java index d06f004fe84c..154554a029fe 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/RemoteGcsHelperTest.java @@ -21,9 +21,11 @@ import com.google.common.collect.ImmutableList; import com.google.gcloud.Page; +import com.google.gcloud.storage.Storage.BlobListOption; import com.google.gcloud.storage.testing.RemoteGcsHelper; import org.easymock.EasyMock; +import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; @@ -66,43 +68,60 @@ public class RemoteGcsHelperTest { + " \"type\": \"service_account\"\n" + "}"; private static final InputStream JSON_KEY_STREAM = new ByteArrayInputStream(JSON_KEY.getBytes()); - private static final List BLOB_LIST = ImmutableList.of( - BlobInfo.builder(BUCKET_NAME, "n1").build(), - BlobInfo.builder(BUCKET_NAME, "n2").build()); private static final StorageException RETRYABLE_EXCEPTION = new StorageException(409, ""); private static final StorageException FATAL_EXCEPTION = new StorageException(500, ""); - private static final Page BLOB_PAGE = new Page() { - @Override - public String nextPageCursor() { - return "nextPageCursor"; - } - - @Override - public Page nextPage() { - return null; - } - - @Override - public Iterable values() { - return BLOB_LIST; - } - - @Override - public Iterator iterateAll() { - return BLOB_LIST.iterator(); - } - }; + private static Storage serviceMockReturnsOptions; + private List blobList; + private Page blobPage; @Rule public ExpectedException thrown = ExpectedException.none(); + @Before + public void setUp() { + serviceMockReturnsOptions = EasyMock.createMock(Storage.class); + EasyMock.expect(serviceMockReturnsOptions.options()) + .andReturn(EasyMock.createMock(StorageOptions.class)) + .times(2); + EasyMock.replay(serviceMockReturnsOptions); + blobList = ImmutableList.of( + new Blob( + serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder(BUCKET_NAME, "n1").build())), + new Blob( + serviceMockReturnsOptions, + new BlobInfo.BuilderImpl(BlobInfo.builder(BUCKET_NAME, "n2").build()))); + blobPage = new Page() { + @Override + public String nextPageCursor() { + return "nextPageCursor"; + } + + @Override + public Page nextPage() { + return null; + } + + @Override + public Iterable values() { + return blobList; + } + + @Override + public Iterator iterateAll() { + return blobList.iterator(); + } + }; + } + @Test public void testForceDelete() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andReturn(true); EasyMock.replay(storageMock); @@ -113,9 +132,9 @@ public void testForceDelete() throws InterruptedException, ExecutionException { @Test public void testForceDeleteTimeout() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE).anyTimes(); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true).anyTimes(); + EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(blobPage).anyTimes(); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true).anyTimes(); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(RETRYABLE_EXCEPTION).anyTimes(); EasyMock.replay(storageMock); @@ -126,9 +145,10 @@ public void testForceDeleteTimeout() throws InterruptedException, ExecutionExcep @Test public void testForceDeleteFail() throws InterruptedException, ExecutionException { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(FATAL_EXCEPTION); EasyMock.replay(storageMock); @@ -143,9 +163,10 @@ public void testForceDeleteFail() throws InterruptedException, ExecutionExceptio @Test public void testForceDeleteNoTimeout() { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andReturn(true); EasyMock.replay(storageMock); @@ -156,9 +177,10 @@ public void testForceDeleteNoTimeout() { @Test public void testForceDeleteNoTimeoutFail() { Storage storageMock = EasyMock.createMock(Storage.class); - EasyMock.expect(storageMock.list(BUCKET_NAME)).andReturn(BLOB_PAGE); - for (BlobInfo info : BLOB_LIST) { - EasyMock.expect(storageMock.delete(BUCKET_NAME, info.name())).andReturn(true); + EasyMock.expect(storageMock.list(BUCKET_NAME, BlobListOption.versions(true))) + .andReturn(blobPage); + for (BlobInfo info : blobList) { + EasyMock.expect(storageMock.delete(info.blobId())).andReturn(true); } EasyMock.expect(storageMock.delete(BUCKET_NAME)).andThrow(FATAL_EXCEPTION); EasyMock.replay(storageMock); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java index 8bef27cb0cd0..c9b957bb936a 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/SerializationTest.java @@ -42,6 +42,7 @@ public class SerializationTest { + private static final Storage STORAGE = StorageOptions.builder().projectId("p").build().service(); private static final Acl.Domain ACL_DOMAIN = new Acl.Domain("domain"); private static final Acl.Group ACL_GROUP = new Acl.Group("group"); private static final Acl.Project ACL_PROJECT_ = new Acl.Project(ProjectRole.VIEWERS, "pid"); @@ -50,16 +51,18 @@ public class SerializationTest { private static final Acl ACL = Acl.of(ACL_DOMAIN, Acl.Role.OWNER); private static final BlobInfo BLOB_INFO = BlobInfo.builder("b", "n").build(); private static final BucketInfo BUCKET_INFO = BucketInfo.of("b"); + private static final Blob BLOB = new Blob(STORAGE, new BlobInfo.BuilderImpl(BLOB_INFO)); + private static final Bucket BUCKET = new Bucket(STORAGE, new BucketInfo.BuilderImpl(BUCKET_INFO)); private static final Cors.Origin ORIGIN = Cors.Origin.any(); private static final Cors CORS = Cors.builder().maxAgeSeconds(1).origins(Collections.singleton(ORIGIN)).build(); private static final BatchRequest BATCH_REQUEST = BatchRequest.builder().delete("B", "N").build(); private static final BatchResponse BATCH_RESPONSE = new BatchResponse( Collections.singletonList(BatchResponse.Result.of(true)), - Collections.>emptyList(), - Collections.>emptyList()); - private static final PageImpl PAGE_RESULT = new PageImpl<>( - null, "c", Collections.singletonList(BlobInfo.builder("b", "n").build())); + Collections.>emptyList(), + Collections.>emptyList()); + private static final PageImpl PAGE_RESULT = + new PageImpl<>(null, "c", Collections.singletonList(BLOB)); private static final Storage.BlobListOption BLOB_LIST_OPTIONS = Storage.BlobListOption.maxResults(100); private static final Storage.BlobSourceOption BLOB_SOURCE_OPTIONS = @@ -96,9 +99,9 @@ public void testServiceOptions() throws Exception { @Test public void testModelAndRequests() throws Exception { Serializable[] objects = {ACL_DOMAIN, ACL_GROUP, ACL_PROJECT_, ACL_USER, ACL_RAW, ACL, - BLOB_INFO, BUCKET_INFO, ORIGIN, CORS, BATCH_REQUEST, BATCH_RESPONSE, PAGE_RESULT, - BLOB_LIST_OPTIONS, BLOB_SOURCE_OPTIONS, BLOB_TARGET_OPTIONS, BUCKET_LIST_OPTIONS, - BUCKET_SOURCE_OPTIONS, BUCKET_TARGET_OPTIONS}; + BLOB_INFO, BLOB, BUCKET_INFO, BUCKET, ORIGIN, CORS, BATCH_REQUEST, BATCH_RESPONSE, + PAGE_RESULT, BLOB_LIST_OPTIONS, BLOB_SOURCE_OPTIONS, BLOB_TARGET_OPTIONS, + BUCKET_LIST_OPTIONS, BUCKET_SOURCE_OPTIONS, BUCKET_TARGET_OPTIONS}; for (Serializable obj : objects) { Object copy = serializeAndDeserialize(obj); assertEquals(obj, obj); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java index f32a51507857..612664de14ae 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/StorageImplTest.java @@ -200,11 +200,14 @@ public class StorageImplTest { Storage.BlobListOption.prefix("prefix"); private static final Storage.BlobListOption BLOB_LIST_FIELDS = Storage.BlobListOption.fields(Storage.BlobField.CONTENT_TYPE, Storage.BlobField.MD5HASH); + private static final Storage.BlobListOption BLOB_LIST_VERSIONS = + Storage.BlobListOption.versions(false); private static final Storage.BlobListOption BLOB_LIST_EMPTY_FIELDS = Storage.BlobListOption.fields(); private static final Map BLOB_LIST_OPTIONS = ImmutableMap.of( StorageRpc.Option.MAX_RESULTS, BLOB_LIST_MAX_RESULT.value(), - StorageRpc.Option.PREFIX, BLOB_LIST_PREFIX.value()); + StorageRpc.Option.PREFIX, BLOB_LIST_PREFIX.value(), + StorageRpc.Option.VERSIONS, BLOB_LIST_VERSIONS.value()); private static final String PRIVATE_KEY_STRING = "MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoG" + "BAL2xolH1zrISQ8+GzOV29BNjjzq4/HIP8Psd1+cZb81vDklSF+95wB250MSE0BDc81pvIMwj5OmIfLg1NY6uB" @@ -238,6 +241,9 @@ public long millis() { private StorageRpc storageRpcMock; private Storage storage; + private Blob expectedBlob1, expectedBlob2, expectedBlob3; + private Bucket expectedBucket1, expectedBucket2; + @Rule public ExpectedException thrown = ExpectedException.none(); @@ -272,10 +278,23 @@ public void tearDown() throws Exception { EasyMock.verify(rpcFactoryMock, storageRpcMock); } + private void initializeService() { + storage = options.service(); + initializeServiceDependentObjects(); + } + + private void initializeServiceDependentObjects() { + expectedBlob1 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO1)); + expectedBlob2 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO2)); + expectedBlob3 = new Blob(storage, new BlobInfo.BuilderImpl(BLOB_INFO3)); + expectedBucket1 = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO1)); + expectedBucket2 = new Bucket(storage, new BucketInfo.BuilderImpl(BUCKET_INFO2)); + } + @Test public void testGetOptions() { EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertSame(options, storage.options()); } @@ -284,9 +303,9 @@ public void testCreateBucket() { EasyMock.expect(storageRpcMock.create(BUCKET_INFO1.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.create(BUCKET_INFO1); - assertEquals(BUCKET_INFO1.toPb(), bucket.toPb()); + initializeService(); + Bucket bucket = storage.create(BUCKET_INFO1); + assertEquals(expectedBucket1, bucket); } @Test @@ -294,10 +313,10 @@ public void testCreateBucketWithOptions() { EasyMock.expect(storageRpcMock.create(BUCKET_INFO1.toPb(), BUCKET_TARGET_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = + initializeService(); + Bucket bucket = storage.create(BUCKET_INFO1, BUCKET_TARGET_METAGENERATION, BUCKET_TARGET_PREDEFINED_ACL); - assertEquals(BUCKET_INFO1, bucket); + assertEquals(expectedBucket1, bucket); } @Test @@ -309,9 +328,9 @@ public void testCreateBlob() throws IOException { EasyMock.eq(EMPTY_RPC_OPTIONS))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(BLOB_INFO1, BLOB_CONTENT); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(BLOB_INFO1, BLOB_CONTENT); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(BLOB_CONTENT.length, byteStream.read(streamBytes)); @@ -332,9 +351,9 @@ public void testCreateEmptyBlob() throws IOException { EasyMock.eq(EMPTY_RPC_OPTIONS))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(BLOB_INFO1); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(BLOB_INFO1); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(-1, byteStream.read(streamBytes)); @@ -353,11 +372,11 @@ public void testCreateBlobWithOptions() throws IOException { EasyMock.eq(BLOB_TARGET_OPTIONS_CREATE))) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.create(BLOB_INFO1, BLOB_CONTENT, BLOB_TARGET_METAGENERATION, BLOB_TARGET_NOT_EXIST, BLOB_TARGET_PREDEFINED_ACL); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); ByteArrayInputStream byteStream = capturedStream.getValue(); byte[] streamBytes = new byte[BLOB_CONTENT.length]; assertEquals(BLOB_CONTENT.length, byteStream.read(streamBytes)); @@ -374,9 +393,9 @@ public void testCreateBlobFromStream() { EasyMock.expect(storageRpcMock.create(infoWithoutHashes.toPb(), fileStream, EMPTY_RPC_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.create(infoWithHashes, fileStream); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.create(infoWithHashes, fileStream); + assertEquals(expectedBlob1, blob); } @Test @@ -384,9 +403,9 @@ public void testGetBucket() { EasyMock.expect(storageRpcMock.get(BucketInfo.of(BUCKET_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1); - assertEquals(BUCKET_INFO1, bucket); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1); + assertEquals(expectedBucket1, bucket); } @Test @@ -394,9 +413,9 @@ public void testGetBucketWithOptions() { EasyMock.expect(storageRpcMock.get(BucketInfo.of(BUCKET_NAME1).toPb(), BUCKET_GET_OPTIONS)) .andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION); - assertEquals(BUCKET_INFO1, bucket); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION); + assertEquals(expectedBucket1, bucket); } @Test @@ -405,8 +424,8 @@ public void testGetBucketWithSelectedFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BucketInfo.of(BUCKET_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_FIELDS); + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_FIELDS); assertEquals(BUCKET_GET_METAGENERATION.value(), capturedOptions.getValue().get(BUCKET_GET_METAGENERATION.rpcOption())); String selector = (String) capturedOptions.getValue().get(BLOB_GET_FIELDS.rpcOption()); @@ -423,8 +442,8 @@ public void testGetBucketWithEmptyFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BucketInfo.of(BUCKET_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BUCKET_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, + initializeService(); + Bucket bucket = storage.get(BUCKET_NAME1, BUCKET_GET_METAGENERATION, BUCKET_GET_EMPTY_FIELDS); assertEquals(BUCKET_GET_METAGENERATION.value(), capturedOptions.getValue().get(BUCKET_GET_METAGENERATION.rpcOption())); @@ -440,9 +459,9 @@ public void testGetBlob() { storageRpcMock.get(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1); + assertEquals(expectedBlob1, blob); } @Test @@ -451,10 +470,10 @@ public void testGetBlobWithOptions() { storageRpcMock.get(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_GET_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -463,10 +482,10 @@ public void testGetBlobWithOptionsFromBlobId() { storageRpcMock.get(BLOB_INFO1.blobId().toPb(), BLOB_GET_OPTIONS)) .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.get(BLOB_INFO1.blobId(), BLOB_GET_METAGENERATION, BLOB_GET_GENERATION_FROM_BLOB_ID); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -475,8 +494,9 @@ public void testGetBlobWithSelectedFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, + initializeService(); + Blob blob = storage.get( + BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION, BLOB_GET_FIELDS); assertEquals(BLOB_GET_METAGENERATION.value(), capturedOptions.getValue().get(BLOB_GET_METAGENERATION.rpcOption())); @@ -488,7 +508,7 @@ public void testGetBlobWithSelectedFields() { assertTrue(selector.contains("contentType")); assertTrue(selector.contains("crc32c")); assertEquals(30, selector.length()); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test @@ -497,8 +517,8 @@ public void testGetBlobWithEmptyFields() { EasyMock.expect(storageRpcMock.get(EasyMock.eq(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb()), EasyMock.capture(capturedOptions))).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, + initializeService(); + Blob blob = storage.get(BUCKET_NAME1, BLOB_NAME1, BLOB_GET_METAGENERATION, BLOB_GET_GENERATION, BLOB_GET_EMPTY_FIELDS); assertEquals(BLOB_GET_METAGENERATION.value(), capturedOptions.getValue().get(BLOB_GET_METAGENERATION.rpcOption())); @@ -508,21 +528,22 @@ public void testGetBlobWithEmptyFields() { assertTrue(selector.contains("bucket")); assertTrue(selector.contains("name")); assertEquals(11, selector.length()); - assertEquals(BLOB_INFO1, blob); + assertEquals(expectedBlob1, blob); } @Test public void testListBuckets() { String cursor = "cursor"; - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test @@ -530,79 +551,84 @@ public void testListBucketsEmpty() { EasyMock.expect(storageRpcMock.list(EMPTY_RPC_OPTIONS)).andReturn( Tuple.>of(null, null)); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(); + initializeService(); + Page page = storage.list(); assertNull(page.nextPageCursor()); - assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(ImmutableList.of().toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithOptions() { String cursor = "cursor"; - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_LIST_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_MAX_RESULT, BUCKET_LIST_PREFIX); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_MAX_RESULT, BUCKET_LIST_PREFIX); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EasyMock.capture(capturedOptions))).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_FIELDS); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_FIELDS); String selector = (String) capturedOptions.getValue().get(BLOB_LIST_FIELDS.rpcOption()); assertTrue(selector.contains("items")); assertTrue(selector.contains("name")); assertTrue(selector.contains("acl")); assertTrue(selector.contains("location")); - assertEquals(24, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(38, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBucketsWithEmptyFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList bucketList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); + ImmutableList bucketInfoList = ImmutableList.of(BUCKET_INFO1, BUCKET_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(bucketList, BucketInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(bucketInfoList, BucketInfo.TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(EasyMock.capture(capturedOptions))).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_LIST_EMPTY_FIELDS); + initializeService(); + ImmutableList bucketList = ImmutableList.of(expectedBucket1, expectedBucket2); + Page page = storage.list(BUCKET_LIST_EMPTY_FIELDS); String selector = (String) capturedOptions.getValue().get(BLOB_LIST_FIELDS.rpcOption()); assertTrue(selector.contains("items")); assertTrue(selector.contains("name")); - assertEquals(11, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(25, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), BucketInfo.class)); + assertArrayEquals(bucketList.toArray(), Iterables.toArray(page.values(), Bucket.class)); } @Test public void testListBlobs() { String cursor = "cursor"; - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_NAME1, EMPTY_RPC_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1); + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test @@ -611,40 +637,42 @@ public void testListBlobsEmpty() { .andReturn(Tuple.>of( null, null)); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1); + initializeService(); + Page page = storage.list(BUCKET_NAME1); assertNull(page.nextPageCursor()); - assertArrayEquals(ImmutableList.of().toArray(), - Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(ImmutableList.of().toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithOptions() { String cursor = "cursor"; - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect(storageRpcMock.list(BUCKET_NAME1, BLOB_LIST_OPTIONS)).andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX); + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = + storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_VERSIONS); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithSelectedFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect( storageRpcMock.list(EasyMock.eq(BUCKET_NAME1), EasyMock.capture(capturedOptions))) .andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_FIELDS); assertEquals(BLOB_LIST_MAX_RESULT.value(), capturedOptions.getValue().get(BLOB_LIST_MAX_RESULT.rpcOption())); @@ -656,24 +684,26 @@ public void testListBlobsWithSelectedFields() { assertTrue(selector.contains("name")); assertTrue(selector.contains("contentType")); assertTrue(selector.contains("md5Hash")); - assertEquals(38, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(52, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test public void testListBlobsWithEmptyFields() { String cursor = "cursor"; Capture> capturedOptions = Capture.newInstance(); - ImmutableList blobList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); + ImmutableList blobInfoList = ImmutableList.of(BLOB_INFO1, BLOB_INFO2); Tuple> result = - Tuple.of(cursor, Iterables.transform(blobList, BlobInfo.TO_PB_FUNCTION)); + Tuple.of(cursor, Iterables.transform(blobInfoList, BlobInfo.INFO_TO_PB_FUNCTION)); EasyMock.expect( storageRpcMock.list(EasyMock.eq(BUCKET_NAME1), EasyMock.capture(capturedOptions))) .andReturn(result); EasyMock.replay(storageRpcMock); - storage = options.service(); - Page page = + initializeService(); + ImmutableList blobList = ImmutableList.of(expectedBlob1, expectedBlob2); + Page page = storage.list(BUCKET_NAME1, BLOB_LIST_MAX_RESULT, BLOB_LIST_PREFIX, BLOB_LIST_EMPTY_FIELDS); assertEquals(BLOB_LIST_MAX_RESULT.value(), capturedOptions.getValue().get(BLOB_LIST_MAX_RESULT.rpcOption())); @@ -683,9 +713,10 @@ public void testListBlobsWithEmptyFields() { assertTrue(selector.contains("items")); assertTrue(selector.contains("bucket")); assertTrue(selector.contains("name")); - assertEquals(18, selector.length()); + assertTrue(selector.contains("nextPageToken")); + assertEquals(32, selector.length()); assertEquals(cursor, page.nextPageCursor()); - assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), BlobInfo.class)); + assertArrayEquals(blobList.toArray(), Iterables.toArray(page.values(), Blob.class)); } @Test @@ -694,9 +725,9 @@ public void testUpdateBucket() { EasyMock.expect(storageRpcMock.patch(updatedBucketInfo.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(updatedBucketInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = storage.update(updatedBucketInfo); - assertEquals(updatedBucketInfo, bucket); + initializeService(); + Bucket bucket = storage.update(updatedBucketInfo); + assertEquals(new Bucket(storage, new BucketInfo.BuilderImpl(updatedBucketInfo)), bucket); } @Test @@ -705,11 +736,11 @@ public void testUpdateBucketWithOptions() { EasyMock.expect(storageRpcMock.patch(updatedBucketInfo.toPb(), BUCKET_TARGET_OPTIONS)) .andReturn(updatedBucketInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BucketInfo bucket = + initializeService(); + Bucket bucket = storage.update(updatedBucketInfo, BUCKET_TARGET_METAGENERATION, BUCKET_TARGET_PREDEFINED_ACL); - assertEquals(updatedBucketInfo, bucket); + assertEquals(new Bucket(storage, new BucketInfo.BuilderImpl(updatedBucketInfo)), bucket); } @Test @@ -718,9 +749,9 @@ public void testUpdateBlob() { EasyMock.expect(storageRpcMock.patch(updatedBlobInfo.toPb(), EMPTY_RPC_OPTIONS)) .andReturn(updatedBlobInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.update(updatedBlobInfo); - assertEquals(updatedBlobInfo, blob); + initializeService(); + Blob blob = storage.update(updatedBlobInfo); + assertEquals(new Blob(storage, new BlobInfo.BuilderImpl(updatedBlobInfo)), blob); } @Test @@ -729,10 +760,10 @@ public void testUpdateBlobWithOptions() { EasyMock.expect(storageRpcMock.patch(updatedBlobInfo.toPb(), BLOB_TARGET_OPTIONS_UPDATE)) .andReturn(updatedBlobInfo.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = + initializeService(); + Blob blob = storage.update(updatedBlobInfo, BLOB_TARGET_METAGENERATION, BLOB_TARGET_PREDEFINED_ACL); - assertEquals(updatedBlobInfo, blob); + assertEquals(new Blob(storage, new BlobInfo.BuilderImpl(updatedBlobInfo)), blob); } @Test @@ -740,7 +771,7 @@ public void testDeleteBucket() { EasyMock.expect(storageRpcMock.delete(BucketInfo.of(BUCKET_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1)); } @@ -750,7 +781,7 @@ public void testDeleteBucketWithOptions() { .expect(storageRpcMock.delete(BucketInfo.of(BUCKET_NAME1).toPb(), BUCKET_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BUCKET_SOURCE_METAGENERATION)); } @@ -760,7 +791,7 @@ public void testDeleteBlob() { storageRpcMock.delete(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BLOB_NAME1)); } @@ -770,7 +801,7 @@ public void testDeleteBlobWithOptions() { storageRpcMock.delete(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BUCKET_NAME1, BLOB_NAME1, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION)); } @@ -781,7 +812,7 @@ public void testDeleteBlobWithOptionsFromBlobId() { storageRpcMock.delete(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(true); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); assertTrue(storage.delete(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION)); } @@ -795,9 +826,9 @@ public void testCompose() { EasyMock.expect(storageRpcMock.compose(ImmutableList.of(BLOB_INFO2.toPb(), BLOB_INFO3.toPb()), BLOB_INFO1.toPb(), EMPTY_RPC_OPTIONS)).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.compose(req); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.compose(req); + assertEquals(expectedBlob1, blob); } @Test @@ -810,9 +841,9 @@ public void testComposeWithOptions() { EasyMock.expect(storageRpcMock.compose(ImmutableList.of(BLOB_INFO2.toPb(), BLOB_INFO3.toPb()), BLOB_INFO1.toPb(), BLOB_TARGET_OPTIONS_COMPOSE)).andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); - storage = options.service(); - BlobInfo blob = storage.compose(req); - assertEquals(BLOB_INFO1, blob); + initializeService(); + Blob blob = storage.compose(req); + assertEquals(expectedBlob1, blob); } @Test @@ -824,7 +855,7 @@ public void testCopy() { false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -844,7 +875,7 @@ public void testCopyWithOptions() { false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -864,7 +895,7 @@ public void testCopyWithOptionsFromBlobId() { new StorageRpc.RewriteResponse(rpcRequest, null, 42L, false, "token", 21L); EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -883,7 +914,7 @@ public void testCopyMultipleRequests() { EasyMock.expect(storageRpcMock.openRewrite(rpcRequest)).andReturn(rpcResponse1); EasyMock.expect(storageRpcMock.continueRewrite(rpcResponse1)).andReturn(rpcResponse2); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); CopyWriter writer = storage.copy(request); assertEquals(42L, writer.blobSize()); assertEquals(21L, writer.totalBytesCopied()); @@ -900,7 +931,7 @@ public void testReadAllBytes() { storageRpcMock.load(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), EMPTY_RPC_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BUCKET_NAME1, BLOB_NAME1); assertArrayEquals(BLOB_CONTENT, readBytes); } @@ -911,7 +942,7 @@ public void testReadAllBytesWithOptions() { storageRpcMock.load(BlobId.of(BUCKET_NAME1, BLOB_NAME1).toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BUCKET_NAME1, BLOB_NAME1, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION); assertArrayEquals(BLOB_CONTENT, readBytes); @@ -923,7 +954,7 @@ public void testReadAllBytesWithOptionsFromBlobId() { storageRpcMock.load(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS)) .andReturn(BLOB_CONTENT); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); byte[] readBytes = storage.readAllBytes(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION); assertArrayEquals(BLOB_CONTENT, readBytes); @@ -970,11 +1001,10 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); BatchResponse batchResponse = storage.submit(req); // Verify captured StorageRpc.BatchRequest @@ -1012,7 +1042,7 @@ public Tuple apply(StorageObject f) { @Test public void testReader() { EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BUCKET_NAME1, BLOB_NAME1); assertNotNull(channel); assertTrue(channel.isOpen()); @@ -1025,7 +1055,7 @@ public void testReaderWithOptions() throws IOException { storageRpcMock.read(BLOB_INFO2.toPb(), BLOB_SOURCE_OPTIONS, 0, DEFAULT_CHUNK_SIZE)) .andReturn(StorageRpc.Tuple.of("etag", result)); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BUCKET_NAME1, BLOB_NAME2, BLOB_SOURCE_GENERATION, BLOB_SOURCE_METAGENERATION); assertNotNull(channel); @@ -1040,7 +1070,7 @@ public void testReaderWithOptionsFromBlobId() throws IOException { storageRpcMock.read(BLOB_INFO1.blobId().toPb(), BLOB_SOURCE_OPTIONS, 0, DEFAULT_CHUNK_SIZE)) .andReturn(StorageRpc.Tuple.of("etag", result)); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); ReadChannel channel = storage.reader(BLOB_INFO1.blobId(), BLOB_SOURCE_GENERATION_FROM_BLOB_ID, BLOB_SOURCE_METAGENERATION); assertNotNull(channel); @@ -1056,7 +1086,7 @@ public void testWriter() { EasyMock.expect(storageRpcMock.open(infoWithoutHashes.toPb(), EMPTY_RPC_OPTIONS)) .andReturn("upload-id"); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); WriteChannel channel = storage.writer(infoWithHashes); assertNotNull(channel); assertTrue(channel.isOpen()); @@ -1068,7 +1098,7 @@ public void testWriterWithOptions() { EasyMock.expect(storageRpcMock.open(info.toPb(), BLOB_TARGET_OPTIONS_CREATE)) .andReturn("upload-id"); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); WriteChannel channel = storage.writer(info, BLOB_WRITE_METAGENERATION, BLOB_WRITE_NOT_EXIST, BLOB_WRITE_PREDEFINED_ACL, BLOB_WRITE_CRC2C, BLOB_WRITE_MD5_HASH); assertNotNull(channel); @@ -1154,12 +1184,11 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); - List resultBlobs = storage.get(blobId1, blobId2); + initializeService(); + List resultBlobs = storage.get(blobId1, blobId2); // Verify captured StorageRpc.BatchRequest List>> capturedToGet = @@ -1197,12 +1226,11 @@ public Tuple apply(StorageObject f) { StorageRpc.BatchResponse res = new StorageRpc.BatchResponse(deleteResult, updateResult, getResult); - Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); - List resultBlobs = storage.update(blobInfo1, blobInfo2); + initializeService(); + List resultBlobs = storage.update(blobInfo1, blobInfo2); // Verify captured StorageRpc.BatchRequest List>> capturedToUpdate = @@ -1243,7 +1271,7 @@ public Tuple apply(StorageObject f) { Capture capturedBatchRequest = Capture.newInstance(); EasyMock.expect(storageRpcMock.batch(EasyMock.capture(capturedBatchRequest))).andReturn(res); EasyMock.replay(storageRpcMock); - storage = options.service(); + initializeService(); List deleteResults = storage.delete(blobInfo1.blobId(), blobInfo2.blobId()); // Verify captured StorageRpc.BatchRequest @@ -1270,8 +1298,9 @@ public void testRetryableException() { .andReturn(BLOB_INFO1.toPb()); EasyMock.replay(storageRpcMock); storage = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); - BlobInfo readBlob = storage.get(blob); - assertEquals(BLOB_INFO1, readBlob); + initializeServiceDependentObjects(); + Blob readBlob = storage.get(blob); + assertEquals(expectedBlob1, readBlob); } @Test @@ -1282,6 +1311,7 @@ public void testNonRetryableException() { .andThrow(new StorageException(501, exceptionMessage)); EasyMock.replay(storageRpcMock); storage = options.toBuilder().retryParams(RetryParams.defaultInstance()).build().service(); + initializeServiceDependentObjects(); thrown.expect(StorageException.class); thrown.expectMessage(exceptionMessage); storage.get(blob); diff --git a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java similarity index 75% rename from gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java rename to gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java index 63b9d739b686..b62a54aff818 100644 --- a/gcloud-java-storage/src/test/java/com/google/gcloud/storage/ITStorageTest.java +++ b/gcloud-java-storage/src/test/java/com/google/gcloud/storage/it/ITStorageTest.java @@ -14,7 +14,7 @@ * limitations under the License. */ -package com.google.gcloud.storage; +package com.google.gcloud.storage.it; import static java.nio.charset.StandardCharsets.UTF_8; import static org.junit.Assert.assertArrayEquals; @@ -25,15 +25,28 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; -import com.google.api.client.util.Lists; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import com.google.gcloud.Page; import com.google.gcloud.ReadChannel; import com.google.gcloud.RestorableState; import com.google.gcloud.WriteChannel; +import com.google.gcloud.storage.BatchRequest; +import com.google.gcloud.storage.BatchResponse; +import com.google.gcloud.storage.Blob; +import com.google.gcloud.storage.BlobId; +import com.google.gcloud.storage.BlobInfo; +import com.google.gcloud.storage.Bucket; +import com.google.gcloud.storage.BucketInfo; +import com.google.gcloud.storage.CopyWriter; +import com.google.gcloud.storage.HttpMethod; +import com.google.gcloud.storage.Storage; import com.google.gcloud.storage.Storage.BlobField; import com.google.gcloud.storage.Storage.BucketField; +import com.google.gcloud.storage.StorageException; import com.google.gcloud.storage.testing.RemoteGcsHelper; import org.junit.AfterClass; @@ -41,6 +54,7 @@ import org.junit.Test; import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.InputStream; import java.net.URL; @@ -52,6 +66,7 @@ import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.logging.Level; @@ -77,8 +92,9 @@ public static void beforeClass() { @AfterClass public static void afterClass() throws ExecutionException, InterruptedException { - if (storage != null && !RemoteGcsHelper.forceDelete(storage, BUCKET, 5, TimeUnit.SECONDS)) { - if (log.isLoggable(Level.WARNING)) { + if (storage != null) { + boolean wasDeleted = RemoteGcsHelper.forceDelete(storage, BUCKET, 5, TimeUnit.SECONDS); + if (!wasDeleted && log.isLoggable(Level.WARNING)) { log.log(Level.WARNING, "Deletion of bucket {0} timed out, bucket is not empty", BUCKET); } } @@ -86,15 +102,15 @@ public static void afterClass() throws ExecutionException, InterruptedException @Test(timeout = 5000) public void testListBuckets() throws InterruptedException { - Iterator bucketIterator = storage.list(Storage.BucketListOption.prefix(BUCKET), - Storage.BucketListOption.fields()).values().iterator(); + Iterator bucketIterator = storage.list(Storage.BucketListOption.prefix(BUCKET), + Storage.BucketListOption.fields()).iterateAll(); while (!bucketIterator.hasNext()) { Thread.sleep(500); bucketIterator = storage.list(Storage.BucketListOption.prefix(BUCKET), - Storage.BucketListOption.fields()).values().iterator(); + Storage.BucketListOption.fields()).iterateAll(); } while (bucketIterator.hasNext()) { - BucketInfo remoteBucket = bucketIterator.next(); + Bucket remoteBucket = bucketIterator.next(); assertTrue(remoteBucket.name().startsWith(BUCKET)); assertNull(remoteBucket.createTime()); assertNull(remoteBucket.selfLink()); @@ -103,7 +119,7 @@ public void testListBuckets() throws InterruptedException { @Test public void testGetBucketSelectedFields() { - BucketInfo remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.ID)); + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.ID)); assertEquals(BUCKET, remoteBucket.name()); assertNull(remoteBucket.createTime()); assertNotNull(remoteBucket.id()); @@ -111,7 +127,7 @@ public void testGetBucketSelectedFields() { @Test public void testGetBucketAllSelectedFields() { - BucketInfo remoteBucket = storage.get(BUCKET, + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields(BucketField.values())); assertEquals(BUCKET, remoteBucket.name()); assertNotNull(remoteBucket.createTime()); @@ -120,7 +136,7 @@ public void testGetBucketAllSelectedFields() { @Test public void testGetBucketEmptyFields() { - BucketInfo remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields()); + Bucket remoteBucket = storage.get(BUCKET, Storage.BucketGetOption.fields()); assertEquals(BUCKET, remoteBucket.name()); assertNull(remoteBucket.createTime()); assertNull(remoteBucket.selfLink()); @@ -130,26 +146,26 @@ public void testGetBucketEmptyFields() { public void testCreateBlob() { String blobName = "test-create-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertArrayEquals(BLOB_BYTE_CONTENT, readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testCreateEmptyBlob() { String blobName = "test-create-empty-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob); + Blob remoteBlob = storage.create(blob); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertArrayEquals(new byte[0], readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -157,21 +173,22 @@ public void testCreateBlobStream() { String blobName = "test-create-blob-stream"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).contentType(CONTENT_TYPE).build(); ByteArrayInputStream stream = new ByteArrayInputStream(BLOB_STRING_CONTENT.getBytes(UTF_8)); - BlobInfo remoteBlob = storage.create(blob, stream); + Blob remoteBlob = storage.create(blob, stream); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); assertEquals(blob.contentType(), remoteBlob.contentType()); byte[] readBytes = storage.readAllBytes(BUCKET, blobName); assertEquals(BLOB_STRING_CONTENT, new String(readBytes, UTF_8)); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testCreateBlobFail() { String blobName = "test-create-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo wrongGenerationBlob = BlobInfo.builder(BUCKET, blobName, -1L).build(); try { storage.create(wrongGenerationBlob, BLOB_BYTE_CONTENT, @@ -180,7 +197,7 @@ public void testCreateBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -204,10 +221,10 @@ public void testGetBlobEmptySelectedFields() { String blobName = "test-get-empty-selected-fields-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).contentType(CONTENT_TYPE).build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields()); + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields()); assertEquals(blob.blobId(), remoteBlob.blobId()); assertNull(remoteBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -218,12 +235,12 @@ public void testGetBlobSelectedFields() { .metadata(ImmutableMap.of("k", "v")) .build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields( + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields( BlobField.METADATA)); assertEquals(blob.blobId(), remoteBlob.blobId()); assertEquals(ImmutableMap.of("k", "v"), remoteBlob.metadata()); assertNull(remoteBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -234,21 +251,22 @@ public void testGetBlobAllSelectedFields() { .metadata(ImmutableMap.of("k", "v")) .build(); assertNotNull(storage.create(blob)); - BlobInfo remoteBlob = storage.get(blob.blobId(), + Blob remoteBlob = storage.get(blob.blobId(), Storage.BlobGetOption.fields(BlobField.values())); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); assertEquals(ImmutableMap.of("k", "v"), remoteBlob.metadata()); assertNotNull(remoteBlob.id()); assertNotNull(remoteBlob.selfLink()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testGetBlobFail() { String blobName = "test-get-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobId wrongGenerationBlob = BlobId.of(BUCKET, blobName); try { storage.get(wrongGenerationBlob, Storage.BlobGetOption.generationMatch(-1)); @@ -256,21 +274,22 @@ public void testGetBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testGetBlobFailNonExistingGeneration() { String blobName = "test-get-blob-fail-non-existing-generation"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobId wrongGenerationBlob = BlobId.of(BUCKET, blobName, -1L); assertNull(storage.get(wrongGenerationBlob)); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } - @Test - public void testListBlobsSelectedFields() { + @Test(timeout = 5000) + public void testListBlobsSelectedFields() throws InterruptedException { String[] blobNames = {"test-list-blobs-selected-fields-blob1", "test-list-blobs-selected-fields-blob2"}; ImmutableMap metadata = ImmutableMap.of("k", "v"); @@ -282,24 +301,36 @@ public void testListBlobsSelectedFields() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob1)); - assertNotNull(storage.create(blob2)); - Page page = storage.list(BUCKET, + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + Page page = storage.list(BUCKET, Storage.BlobListOption.prefix("test-list-blobs-selected-fields-blob"), Storage.BlobListOption.fields(BlobField.METADATA)); - int index = 0; - for (BlobInfo remoteBlob : page.values()) { + // Listing blobs is eventually consistent, we loop until the list is of the expected size. The + // test fails if timeout is reached. + while (Iterators.size(page.iterateAll()) != 2) { + Thread.sleep(500); + page = storage.list(BUCKET, + Storage.BlobListOption.prefix("test-list-blobs-selected-fields-blob"), + Storage.BlobListOption.fields(BlobField.METADATA)); + } + Set blobSet = ImmutableSet.of(blobNames[0], blobNames[1]); + Iterator iterator = page.iterateAll(); + while (iterator.hasNext()) { + Blob remoteBlob = iterator.next(); assertEquals(BUCKET, remoteBlob.bucket()); - assertEquals(blobNames[index++], remoteBlob.name()); + assertTrue(blobSet.contains(remoteBlob.name())); assertEquals(metadata, remoteBlob.metadata()); assertNull(remoteBlob.contentType()); } - assertTrue(storage.delete(BUCKET, blobNames[0])); - assertTrue(storage.delete(BUCKET, blobNames[1])); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); } - @Test - public void testListBlobsEmptySelectedFields() { + @Test(timeout = 5000) + public void testListBlobsEmptySelectedFields() throws InterruptedException { String[] blobNames = {"test-list-blobs-empty-selected-fields-blob1", "test-list-blobs-empty-selected-fields-blob2"}; BlobInfo blob1 = BlobInfo.builder(BUCKET, blobNames[0]) @@ -308,32 +339,90 @@ public void testListBlobsEmptySelectedFields() { BlobInfo blob2 = BlobInfo.builder(BUCKET, blobNames[1]) .contentType(CONTENT_TYPE) .build(); - assertNotNull(storage.create(blob1)); - assertNotNull(storage.create(blob2)); - Page page = storage.list(BUCKET, + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + Page page = storage.list(BUCKET, Storage.BlobListOption.prefix("test-list-blobs-empty-selected-fields-blob"), Storage.BlobListOption.fields()); - int index = 0; - for (BlobInfo remoteBlob : page.values()) { + // Listing blobs is eventually consistent, we loop until the list is of the expected size. The + // test fails if timeout is reached. + while (Iterators.size(page.iterateAll()) != 2) { + Thread.sleep(500); + page = storage.list(BUCKET, + Storage.BlobListOption.prefix("test-list-blobs-empty-selected-fields-blob"), + Storage.BlobListOption.fields()); + } + Set blobSet = ImmutableSet.of(blobNames[0], blobNames[1]); + Iterator iterator = page.iterateAll(); + while (iterator.hasNext()) { + Blob remoteBlob = iterator.next(); assertEquals(BUCKET, remoteBlob.bucket()); - assertEquals(blobNames[index++], remoteBlob.name()); + assertTrue(blobSet.contains(remoteBlob.name())); assertNull(remoteBlob.contentType()); } - assertTrue(storage.delete(BUCKET, blobNames[0])); - assertTrue(storage.delete(BUCKET, blobNames[1])); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); + } + + @Test(timeout = 15000) + public void testListBlobsVersioned() throws ExecutionException, InterruptedException { + String bucketName = RemoteGcsHelper.generateBucketName(); + Bucket bucket = storage.create(BucketInfo.builder(bucketName).versioningEnabled(true).build()); + try { + String[] blobNames = {"test-list-blobs-versioned-blob1", "test-list-blobs-versioned-blob2"}; + BlobInfo blob1 = BlobInfo.builder(bucket, blobNames[0]) + .contentType(CONTENT_TYPE) + .build(); + BlobInfo blob2 = BlobInfo.builder(bucket, blobNames[1]) + .contentType(CONTENT_TYPE) + .build(); + Blob remoteBlob1 = storage.create(blob1); + Blob remoteBlob2 = storage.create(blob2); + Blob remoteBlob3 = storage.create(blob2); + assertNotNull(remoteBlob1); + assertNotNull(remoteBlob2); + assertNotNull(remoteBlob3); + Page page = storage.list(bucketName, + Storage.BlobListOption.prefix("test-list-blobs-versioned-blob"), + Storage.BlobListOption.versions(true)); + // Listing blobs is eventually consistent, we loop until the list is of the expected size. The + // test fails if timeout is reached. + while (Iterators.size(page.iterateAll()) != 3) { + Thread.sleep(500); + page = storage.list(bucketName, + Storage.BlobListOption.prefix("test-list-blobs-versioned-blob"), + Storage.BlobListOption.versions(true)); + } + Set blobSet = ImmutableSet.of(blobNames[0], blobNames[1]); + Iterator iterator = page.iterateAll(); + while (iterator.hasNext()) { + Blob remoteBlob = iterator.next(); + assertEquals(bucketName, remoteBlob.bucket()); + assertTrue(blobSet.contains(remoteBlob.name())); + assertNotNull(remoteBlob.generation()); + } + assertTrue(remoteBlob1.delete()); + assertTrue(remoteBlob2.delete()); + assertTrue(remoteBlob3.delete()); + } finally { + RemoteGcsHelper.forceDelete(storage, bucketName, 5, TimeUnit.SECONDS); + } } @Test public void testUpdateBlob() { String blobName = "test-update-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().contentType(CONTENT_TYPE).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().contentType(CONTENT_TYPE).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(CONTENT_TYPE, updatedBlob.contentType()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -345,15 +434,16 @@ public void testUpdateBlobReplaceMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(null).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(null).build().update(); assertNotNull(updatedBlob); assertNull(updatedBlob.metadata()); - updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(newMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -366,13 +456,14 @@ public void testUpdateBlobMergeMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(expectedMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test @@ -387,20 +478,22 @@ public void testUpdateBlobUnsetMetadata() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob)); - BlobInfo updatedBlob = storage.update(blob.toBuilder().metadata(newMetadata).build()); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); + Blob updatedBlob = remoteBlob.toBuilder().metadata(newMetadata).build().update(); assertNotNull(updatedBlob); assertEquals(blob.name(), updatedBlob.name()); assertEquals(blob.bucket(), updatedBlob.bucket()); assertEquals(expectedMetadata, updatedBlob.metadata()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(updatedBlob.delete()); } @Test public void testUpdateBlobFail() { String blobName = "test-update-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo wrongGenerationBlob = BlobInfo.builder(BUCKET, blobName, -1L) .contentType(CONTENT_TYPE) .build(); @@ -410,13 +503,13 @@ public void testUpdateBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test public void testDeleteNonExistingBlob() { String blobName = "test-delete-non-existing-blob"; - assertTrue(!storage.delete(BUCKET, blobName)); + assertFalse(storage.delete(BUCKET, blobName)); } @Test @@ -424,21 +517,22 @@ public void testDeleteBlobNonExistingGeneration() { String blobName = "test-delete-blob-non-existing-generation"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); assertNotNull(storage.create(blob)); - assertTrue(!storage.delete(BlobId.of(BUCKET, blobName, -1L))); + assertFalse(storage.delete(BlobId.of(BUCKET, blobName, -1L))); } @Test public void testDeleteBlobFail() { String blobName = "test-delete-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); try { storage.delete(BUCKET, blob.name(), Storage.BlobSourceOption.generationMatch(-1L)); fail("StorageException was expected"); } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blob.name())); + assertTrue(remoteBlob.delete()); } @Test @@ -447,24 +541,26 @@ public void testComposeBlob() { String sourceBlobName2 = "test-compose-blob-source-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - assertNotNull(storage.create(sourceBlob1, BLOB_BYTE_CONTENT)); - assertNotNull(storage.create(sourceBlob2, BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob1 = storage.create(sourceBlob1, BLOB_BYTE_CONTENT); + Blob remoteSourceBlob2 = storage.create(sourceBlob2, BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob1); + assertNotNull(remoteSourceBlob2); String targetBlobName = "test-compose-blob-target"; BlobInfo targetBlob = BlobInfo.builder(BUCKET, targetBlobName).build(); Storage.ComposeRequest req = Storage.ComposeRequest.of(ImmutableList.of(sourceBlobName1, sourceBlobName2), targetBlob); - BlobInfo remoteBlob = storage.compose(req); - assertNotNull(remoteBlob); - assertEquals(targetBlob.name(), remoteBlob.name()); - assertEquals(targetBlob.bucket(), remoteBlob.bucket()); + Blob remoteTargetBlob = storage.compose(req); + assertNotNull(remoteTargetBlob); + assertEquals(targetBlob.name(), remoteTargetBlob.name()); + assertEquals(targetBlob.bucket(), remoteTargetBlob.bucket()); byte[] readBytes = storage.readAllBytes(BUCKET, targetBlobName); byte[] composedBytes = Arrays.copyOf(BLOB_BYTE_CONTENT, BLOB_BYTE_CONTENT.length * 2); System.arraycopy(BLOB_BYTE_CONTENT, 0, composedBytes, BLOB_BYTE_CONTENT.length, BLOB_BYTE_CONTENT.length); assertArrayEquals(composedBytes, readBytes); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); - assertTrue(storage.delete(BUCKET, targetBlobName)); + assertTrue(remoteSourceBlob1.delete()); + assertTrue(remoteSourceBlob2.delete()); + assertTrue(remoteTargetBlob.delete()); } @Test @@ -473,8 +569,10 @@ public void testComposeBlobFail() { String sourceBlobName2 = "test-compose-blob-fail-source-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - assertNotNull(storage.create(sourceBlob1)); - assertNotNull(storage.create(sourceBlob2)); + Blob remoteSourceBlob1 = storage.create(sourceBlob1); + Blob remoteSourceBlob2 = storage.create(sourceBlob2); + assertNotNull(remoteSourceBlob1); + assertNotNull(remoteSourceBlob2); String targetBlobName = "test-compose-blob-fail-target"; BlobInfo targetBlob = BlobInfo.builder(BUCKET, targetBlobName).build(); Storage.ComposeRequest req = Storage.ComposeRequest.builder() @@ -488,8 +586,8 @@ public void testComposeBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteSourceBlob1.delete()); + assertTrue(remoteSourceBlob2.delete()); } @Test @@ -501,7 +599,8 @@ public void testCopyBlob() { .contentType(CONTENT_TYPE) .metadata(metadata) .build(); - assertNotNull(storage.create(blob, BLOB_BYTE_CONTENT)); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + assertNotNull(remoteBlob); String targetBlobName = "test-copy-blob-target"; Storage.CopyRequest req = Storage.CopyRequest.of(source, BlobId.of(BUCKET, targetBlobName)); CopyWriter copyWriter = storage.copy(req); @@ -510,7 +609,7 @@ public void testCopyBlob() { assertEquals(CONTENT_TYPE, copyWriter.result().contentType()); assertEquals(metadata, copyWriter.result().metadata()); assertTrue(copyWriter.isDone()); - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteBlob.delete()); assertTrue(storage.delete(BUCKET, targetBlobName)); } @@ -518,7 +617,8 @@ public void testCopyBlob() { public void testCopyBlobUpdateMetadata() { String sourceBlobName = "test-copy-blob-update-metadata-source"; BlobId source = BlobId.of(BUCKET, sourceBlobName); - assertNotNull(storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob = storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob); String targetBlobName = "test-copy-blob-update-metadata-target"; ImmutableMap metadata = ImmutableMap.of("k", "v"); BlobInfo target = BlobInfo.builder(BUCKET, targetBlobName) @@ -532,7 +632,7 @@ public void testCopyBlobUpdateMetadata() { assertEquals(CONTENT_TYPE, copyWriter.result().contentType()); assertEquals(metadata, copyWriter.result().metadata()); assertTrue(copyWriter.isDone()); - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteSourceBlob.delete()); assertTrue(storage.delete(BUCKET, targetBlobName)); } @@ -540,7 +640,8 @@ public void testCopyBlobUpdateMetadata() { public void testCopyBlobFail() { String sourceBlobName = "test-copy-blob-source-fail"; BlobId source = BlobId.of(BUCKET, sourceBlobName, -1L); - assertNotNull(storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT)); + Blob remoteSourceBlob = storage.create(BlobInfo.builder(source).build(), BLOB_BYTE_CONTENT); + assertNotNull(remoteSourceBlob); String targetBlobName = "test-copy-blob-target-fail"; BlobInfo target = BlobInfo.builder(BUCKET, targetBlobName).contentType(CONTENT_TYPE).build(); Storage.CopyRequest req = Storage.CopyRequest.builder() @@ -565,7 +666,7 @@ public void testCopyBlobFail() { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, sourceBlobName)); + assertTrue(remoteSourceBlob.delete()); } @Test @@ -658,25 +759,26 @@ public void testBatchRequestManyDeletes() { } // Check updates - BlobInfo remoteUpdatedBlob2 = response.updates().get(0).get(); + Blob remoteUpdatedBlob2 = response.updates().get(0).get(); assertEquals(sourceBlob2.bucket(), remoteUpdatedBlob2.bucket()); assertEquals(sourceBlob2.name(), remoteUpdatedBlob2.name()); assertEquals(updatedBlob2.contentType(), remoteUpdatedBlob2.contentType()); // Check gets - BlobInfo remoteBlob1 = response.gets().get(0).get(); + Blob remoteBlob1 = response.gets().get(0).get(); assertEquals(sourceBlob1.bucket(), remoteBlob1.bucket()); assertEquals(sourceBlob1.name(), remoteBlob1.name()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteBlob1.delete()); + assertTrue(remoteUpdatedBlob2.delete()); } @Test public void testBatchRequestFail() { String blobName = "test-batch-request-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); BlobInfo updatedBlob = BlobInfo.builder(BUCKET, blobName, -1L).build(); BatchRequest batchRequest = BatchRequest.builder() .update(updatedBlob, Storage.BlobTargetOption.generationMatch()) @@ -696,7 +798,7 @@ public void testBatchRequestFail() { assertTrue(batchResponse.deletes().get(0).failed()); assertFalse(batchResponse.deletes().get(1).failed()); assertFalse(batchResponse.deletes().get(1).get()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -722,6 +824,33 @@ public void testReadAndWriteChannels() throws IOException { assertTrue(storage.delete(BUCKET, blobName)); } + @Test + public void testReadAndWriteChannelsWithDifferentFileSize() throws IOException { + String blobNamePrefix = "test-read-and-write-channels-blob-"; + int[] blobSizes = {0, 700, 1024 * 256, 2 * 1024 * 1024, 4 * 1024 * 1024, 4 * 1024 * 1024 + 1}; + Random rnd = new Random(); + for (int blobSize : blobSizes) { + String blobName = blobNamePrefix + blobSize; + BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); + byte[] bytes = new byte[blobSize]; + rnd.nextBytes(bytes); + try (WriteChannel writer = storage.writer(blob)) { + writer.write(ByteBuffer.wrap(bytes)); + } + ByteArrayOutputStream output = new ByteArrayOutputStream(); + try (ReadChannel reader = storage.reader(blob.blobId())) { + ByteBuffer buffer = ByteBuffer.allocate(64 * 1024); + while (reader.read(buffer) > 0) { + buffer.flip(); + output.write(buffer.array(), 0, buffer.limit()); + buffer.clear(); + } + } + assertArrayEquals(bytes, output.toByteArray()); + assertTrue(storage.delete(BUCKET, blobName)); + } + } + @Test public void testReadAndWriteCaptureChannels() throws IOException { String blobName = "test-read-and-write-capture-channels-blob"; @@ -755,7 +884,8 @@ public void testReadAndWriteCaptureChannels() throws IOException { public void testReadChannelFail() throws IOException { String blobName = "test-read-channel-blob-fail"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob)); + Blob remoteBlob = storage.create(blob); + assertNotNull(remoteBlob); try (ReadChannel reader = storage.reader(blob.blobId(), Storage.BlobSourceOption.metagenerationMatch(-1L))) { reader.read(ByteBuffer.allocate(42)); @@ -778,7 +908,7 @@ public void testReadChannelFail() throws IOException { } catch (StorageException ex) { // expected } - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -790,7 +920,7 @@ public void testReadChannelFailUpdatedGeneration() throws IOException { int blobSize = 2 * chunkSize; byte[] content = new byte[blobSize]; random.nextBytes(content); - BlobInfo remoteBlob = storage.create(blob, content); + Blob remoteBlob = storage.create(blob, content); assertNotNull(remoteBlob); assertEquals(blobSize, (long) remoteBlob.size()); try (ReadChannel reader = storage.reader(blob.blobId())) { @@ -834,9 +964,9 @@ public void testWriteChannelFail() throws IOException { public void testWriteChannelExistingBlob() throws IOException { String blobName = "test-write-channel-existing-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - BlobInfo remoteBlob = storage.create(blob); + storage.create(blob); byte[] stringBytes; - try (WriteChannel writer = storage.writer(remoteBlob)) { + try (WriteChannel writer = storage.writer(blob)) { stringBytes = BLOB_STRING_CONTENT.getBytes(UTF_8); writer.write(ByteBuffer.wrap(stringBytes)); } @@ -848,14 +978,15 @@ public void testWriteChannelExistingBlob() throws IOException { public void testGetSignedUrl() throws IOException { String blobName = "test-get-signed-url-blob"; BlobInfo blob = BlobInfo.builder(BUCKET, blobName).build(); - assertNotNull(storage.create(blob, BLOB_BYTE_CONTENT)); + Blob remoteBlob = storage.create(blob, BLOB_BYTE_CONTENT); + assertNotNull(remoteBlob); URL url = storage.signUrl(blob, 1, TimeUnit.HOURS); URLConnection connection = url.openConnection(); byte[] readBytes = new byte[BLOB_BYTE_CONTENT.length]; try (InputStream responseStream = connection.getInputStream()) { assertEquals(BLOB_BYTE_CONTENT.length, responseStream.read(readBytes)); assertArrayEquals(BLOB_BYTE_CONTENT, readBytes); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } } @@ -869,11 +1000,11 @@ public void testPostSignedUrl() throws IOException { URLConnection connection = url.openConnection(); connection.setDoOutput(true); connection.connect(); - BlobInfo remoteBlob = storage.get(BUCKET, blobName); + Blob remoteBlob = storage.get(BUCKET, blobName); assertNotNull(remoteBlob); assertEquals(blob.bucket(), remoteBlob.bucket()); assertEquals(blob.name(), remoteBlob.name()); - assertTrue(storage.delete(BUCKET, blobName)); + assertTrue(remoteBlob.delete()); } @Test @@ -884,13 +1015,13 @@ public void testGetBlobs() { BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); assertNotNull(storage.create(sourceBlob1)); assertNotNull(storage.create(sourceBlob2)); - List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); + List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); assertEquals(sourceBlob1.bucket(), remoteBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), remoteBlobs.get(0).name()); assertEquals(sourceBlob2.bucket(), remoteBlobs.get(1).bucket()); assertEquals(sourceBlob2.name(), remoteBlobs.get(1).name()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(remoteBlobs.get(0).delete()); + assertTrue(remoteBlobs.get(1).delete()); } @Test @@ -900,11 +1031,11 @@ public void testGetBlobsFail() { BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); assertNotNull(storage.create(sourceBlob1)); - List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); + List remoteBlobs = storage.get(sourceBlob1.blobId(), sourceBlob2.blobId()); assertEquals(sourceBlob1.bucket(), remoteBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), remoteBlobs.get(0).name()); assertNull(remoteBlobs.get(1)); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); + assertTrue(remoteBlobs.get(0).delete()); } @Test @@ -929,7 +1060,7 @@ public void testDeleteBlobsFail() { assertNotNull(storage.create(sourceBlob1)); List deleteStatus = storage.delete(sourceBlob1.blobId(), sourceBlob2.blobId()); assertTrue(deleteStatus.get(0)); - assertTrue(!deleteStatus.get(1)); + assertFalse(deleteStatus.get(1)); } @Test @@ -938,11 +1069,11 @@ public void testUpdateBlobs() { String sourceBlobName2 = "test-update-blobs-2"; BlobInfo sourceBlob1 = BlobInfo.builder(BUCKET, sourceBlobName1).build(); BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); - BlobInfo remoteBlob1 = storage.create(sourceBlob1); - BlobInfo remoteBlob2 = storage.create(sourceBlob2); + Blob remoteBlob1 = storage.create(sourceBlob1); + Blob remoteBlob2 = storage.create(sourceBlob2); assertNotNull(remoteBlob1); assertNotNull(remoteBlob2); - List updatedBlobs = storage.update( + List updatedBlobs = storage.update( remoteBlob1.toBuilder().contentType(CONTENT_TYPE).build(), remoteBlob2.toBuilder().contentType(CONTENT_TYPE).build()); assertEquals(sourceBlob1.bucket(), updatedBlobs.get(0).bucket()); @@ -951,8 +1082,8 @@ public void testUpdateBlobs() { assertEquals(sourceBlob2.bucket(), updatedBlobs.get(1).bucket()); assertEquals(sourceBlob2.name(), updatedBlobs.get(1).name()); assertEquals(CONTENT_TYPE, updatedBlobs.get(1).contentType()); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); - assertTrue(storage.delete(BUCKET, sourceBlobName2)); + assertTrue(updatedBlobs.get(0).delete()); + assertTrue(updatedBlobs.get(1).delete()); } @Test @@ -963,13 +1094,13 @@ public void testUpdateBlobsFail() { BlobInfo sourceBlob2 = BlobInfo.builder(BUCKET, sourceBlobName2).build(); BlobInfo remoteBlob1 = storage.create(sourceBlob1); assertNotNull(remoteBlob1); - List updatedBlobs = storage.update( + List updatedBlobs = storage.update( remoteBlob1.toBuilder().contentType(CONTENT_TYPE).build(), sourceBlob2.toBuilder().contentType(CONTENT_TYPE).build()); assertEquals(sourceBlob1.bucket(), updatedBlobs.get(0).bucket()); assertEquals(sourceBlob1.name(), updatedBlobs.get(0).name()); assertEquals(CONTENT_TYPE, updatedBlobs.get(0).contentType()); assertNull(updatedBlobs.get(1)); - assertTrue(storage.delete(BUCKET, sourceBlobName1)); + assertTrue(updatedBlobs.get(0).delete()); } } diff --git a/gcloud-java/README.md b/gcloud-java/README.md index a080d787d6ab..e296d0c0c565 100644 --- a/gcloud-java/README.md +++ b/gcloud-java/README.md @@ -6,6 +6,8 @@ Java idiomatic client for [Google Cloud Platform][cloud-platform] services. [![Build Status](https://travis-ci.org/GoogleCloudPlatform/gcloud-java.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/gcloud-java) [![Coverage Status](https://coveralls.io/repos/GoogleCloudPlatform/gcloud-java/badge.svg?branch=master)](https://coveralls.io/r/GoogleCloudPlatform/gcloud-java?branch=master) [![Maven](https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg)]( https://img.shields.io/maven-central/v/com.google.gcloud/gcloud-java.svg) +[![Codacy Badge](https://api.codacy.com/project/badge/grade/9da006ad7c3a4fe1abd142e77c003917)](https://www.codacy.com/app/mziccard/gcloud-java) +[![Dependency Status](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969/badge.svg?style=flat)](https://www.versioneye.com/user/projects/56bd8ee72a29ed002d2b0969) - [Homepage] (https://googlecloudplatform.github.io/gcloud-java/) - [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs) @@ -25,16 +27,16 @@ If you are using Maven, add this to your pom.xml file com.google.gcloud gcloud-java - 0.1.3 + 0.1.5 ``` If you are using Gradle, add this to your dependencies ```Groovy -compile 'com.google.gcloud:gcloud-java:0.1.3' +compile 'com.google.gcloud:gcloud-java:0.1.5' ``` If you are using SBT, add this to your dependencies ```Scala -libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.3" +libraryDependencies += "com.google.gcloud" % "gcloud-java" % "0.1.5" ``` Troubleshooting diff --git a/gcloud-java/pom.xml b/gcloud-java/pom.xml index 60e98fa63396..19536faa8c8d 100644 --- a/gcloud-java/pom.xml +++ b/gcloud-java/pom.xml @@ -1,7 +1,6 @@ 4.0.0 - com.google.gcloud gcloud-java jar GCloud Java @@ -11,7 +10,7 @@ com.google.gcloud gcloud-java-pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT diff --git a/pom.xml b/pom.xml index 2d1bfb1e0a74..b9d979c4d467 100644 --- a/pom.xml +++ b/pom.xml @@ -4,13 +4,31 @@ com.google.gcloud gcloud-java-pom pom - 0.1.4-SNAPSHOT + 0.1.6-SNAPSHOT GCloud Java https://github.com/GoogleCloudPlatform/gcloud-java Java idiomatic client for Google Cloud Platform services. + + derka + Martin Derka + derka@google.com + Google + + Developer + + + + ajaykannan + Ajay Kannan + ajaykannan@google.com + Google + + Developer + + ozarov Arie Ozarov @@ -20,6 +38,15 @@ Developer + + mziccardi + Marco Ziccardi + mziccardi@google.com + Google + + Developer + + Google @@ -106,7 +133,7 @@ org.apache.maven.plugins maven-surefire-plugin - 2.18 + 2.19.1 @@ -117,7 +144,7 @@ org.apache.maven.plugins maven-enforcer-plugin - 1.4 + 1.4.1 enforce-maven @@ -159,7 +186,7 @@ org.apache.maven.plugins maven-failsafe-plugin - 2.18.1 + 2.19.1 @@ -191,7 +218,7 @@ maven-compiler-plugin - 3.2 + 3.5.1 1.7 1.7 @@ -202,7 +229,7 @@ org.apache.maven.plugins maven-source-plugin - 2.4 + 3.0.0 attach-sources @@ -242,7 +269,7 @@ org.sonatype.plugins nexus-staging-maven-plugin - 1.6.5 + 1.6.6 true sonatype-nexus-staging @@ -253,7 +280,7 @@ org.eluder.coveralls coveralls-maven-plugin - 3.1.0 + 4.1.0 ${basedir}/target/coverage.xml @@ -286,12 +313,12 @@ org.apache.maven.plugins maven-checkstyle-plugin - 2.16 + 2.17 com.puppycrawl.tools checkstyle - 6.8.1 + 6.15 @@ -310,7 +337,7 @@ org.apache.maven.plugins maven-project-info-reports-plugin - 2.8 + 2.8.1 @@ -367,12 +394,12 @@ org.apache.maven.plugins maven-surefire-report-plugin - 2.18.1 + 2.19.1 org.apache.maven.plugins maven-checkstyle-plugin - 2.16 + 2.17 checkstyle.xml false diff --git a/src/site/resources/index.html b/src/site/resources/index.html index 8f40cfbcd97e..0e0933e7b68c 100644 --- a/src/site/resources/index.html +++ b/src/site/resources/index.html @@ -178,8 +178,10 @@

    Examples

    • - SparkJava demo - Use gcloud-java with App Engine Managed VMs, Datastore, and SparkJava. + SparkJava demo - Uses gcloud-java with App Engine Managed VMs, Datastore, and SparkJava.
    • +
    • + Bookshelf - An App Engine app that manages a virtual bookshelf using gcloud-java libraries for Datastore and Storage.
    diff --git a/src/site/site.xml b/src/site/site.xml index 55047ce85c54..6279179eb389 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -20,7 +20,7 @@ org.apache.maven.skins maven-fluido-skin - 1.3.1 + 1.4