[May 25, 2023] New 2023 Databricks Associate-Developer-Apache-Spark Exam Dumps with PDF from TrainingQuiz (Updated 179 Questions) [Q10-Q31]

NEW QUESTION 10
The code block shown below should read all files with the file ending .png in directory path into Spark.
Choose the answer that correctly fills the blanks in the code block to accomplish this.
spark.__1__.__2__(__3__).option(__4__, “*.png”).__5__(path)

1. read()
2. format
3. “binaryFile”
4. “recursiveFileLookup”
5. load

1. read
2. format
3. “binaryFile”
4. “pathGlobFilter”
5. load

1. read
2. format
3. binaryFile
4. pathGlobFilter
5. load

1. open
2. format
3. “image”
4. “fileType”
5. open

1. open
2. as
3. “binaryFile”
4. “pathGlobFilter”
5. load

NEW QUESTION 11
Which of the following code blocks generally causes a great amount of network traffic?

DataFrame.select()

DataFrame.coalesce()

DataFrame.collect()

DataFrame.rdd.map()

DataFrame.count()

NEW QUESTION 12
The code block shown below should return only the average prediction error (column predError) of a random subset, without replacement, of approximately 15% of rows in DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__, __3__).__4__(avg(‘predError’))

1. sample
2. True
3. 0.15
4. filter

1. sample
2. False
3. 0.15
4. select

1. sample
2. 0.85
3. False
4. select

1. fraction
2. 0.15
3. True
4. where

1. fraction
2. False
3. 0.85
4. select

NEW QUESTION 13
Which of the following is a characteristic of the cluster manager?

Each cluster manager works on a single partition of data.

The cluster manager receives input from the driver through the SparkContext.

The cluster manager does not exist in standalone mode.

The cluster manager transforms jobs into DAGs.

In client mode, the cluster manager runs on the edge node.

NEW QUESTION 14
Which of the following describes Spark’s Adaptive Query Execution?

Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

Adaptive Query Execution is enabled in Spark by default.

Adaptive Query Execution reoptimizes queries at execution points.

Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.

Adaptive Query Execution applies to all kinds of queries.

NEW QUESTION 15
The code block displayed below contains an error. The code block should display the schema of DataFrame transactionsDf. Find the error.
Code block:
transactionsDf.rdd.printSchema

There is no way to print a schema directly in Spark, since the schema can be printed easily through using print(transactionsDf.columns), so that should be used instead.

The code block should be wrapped into a print() operation.

printSchema is only accessible through the spark session, so the code block should be rewritten as spark.printSchema(transactionsDf).

printSchema is a method and should be written as printSchema(). It is also not callable through transactionsDf.rdd, but should be called directly from transactionsDf.
(Correct)

printSchema is a not a method of transactionsDf.rdd. Instead, the schema should be printed via transactionsDf.print_schema().

NEW QUESTION 16
The code block shown below should add a column itemNameBetweenSeparators to DataFrame itemsDf. The column should contain arrays of maximum 4 strings. The arrays should be composed of the values in column itemsDf which are separated at – or whitespace characters. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Sample of DataFrame itemsDf:
1.+——+———————————-+——————-+
2.|itemId|itemName |supplier |
3.+——+———————————-+——————-+
4.|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|
5.|2 |Elegant Outdoors Summer Dress |YetiX |
6.|3 |Outdoors Backpack |Sports Company Inc.|
7.+——+———————————-+——————-+
Code block:
itemsDf.__1__(__2__, __3__(__4__, “[s-]”, __5__))

1. withColumn
2. “itemNameBetweenSeparators”
3. split
4. “itemName”
5. 4
(Correct)

1. withColumnRenamed
2. “itemNameBetweenSeparators”
3. split
4. “itemName”
5. 4

1. withColumnRenamed
2. “itemName”
3. split
4. “itemNameBetweenSeparators”
5. 4

1. withColumn
2. “itemNameBetweenSeparators”
3. split
4. “itemName”
5. 5

1. withColumn
2. itemNameBetweenSeparators
3. str_split
4. “itemName”
5. 5

NEW QUESTION 17
Which of the following describes Spark actions?

Writing data to disk is the primary purpose of actions.

Actions are Spark’s way of exchanging data between executors.

The driver receives data upon request by actions.

Stage boundaries are commonly established by actions.

Actions are Spark’s way of modifying RDDs.

NEW QUESTION 18
The code block displayed below contains one or more errors. The code block should load parquet files at location filePath into a DataFrame, only loading those files that have been modified before
2029-03-20 05:44:46. Spark should enforce a schema according to the schema shown below. Find the error.
Schema:
1.root
2. |– itemId: integer (nullable = true)
3. |– attributes: array (nullable = true)
4. | |– element: string (containsNull = true)
5. |– supplier: string (nullable = true)
Code block:
1.schema = StructType([
2. StructType(“itemId”, IntegerType(), True),
3. StructType(“attributes”, ArrayType(StringType(), True), True),
4. StructType(“supplier”, StringType(), True)
5.])
6.
7.spark.read.options(“modifiedBefore”, “2029-03-20T05:44:46”).schema(schema).load(filePath)

The attributes array is specified incorrectly, Spark cannot identify the file format, and the syntax of the call to Spark’s DataFrameReader is incorrect.

Columns in the schema definition use the wrong object type and the syntax of the call to Spark’s DataFrameReader is incorrect.

The data type of the schema is incompatible with the schema() operator and the modification date threshold is specified incorrectly.

Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.

Columns in the schema are unable to handle empty values and the modification date threshold is specified incorrectly.

Explanation
Correct code block:
schema = StructType([
StructField(“itemId”, IntegerType(), True),
StructField(“attributes”, ArrayType(StringType(), True), True),
StructField(“supplier”, StringType(), True)
])
spark.read.options(modifiedBefore=”2029-03-20T05:44:46″).schema(schema).parquet(filePath) This question is more difficult than what you would encounter in the exam. In the exam, for this question type, only one error needs to be identified and not “one or multiple” as in the question.
Columns in the schema definition use the wrong object type, the modification date threshold is specified incorrectly, and Spark cannot identify the file format.
Correct! Columns in the schema definition should use the StructField type. Building a schema from pyspark.sql.types, as here using classes like StructType and StructField, is one of multiple ways of expressing a schema in Spark. A StructType always contains a list of StructFields (see documentation linked below). So, nesting StructType and StructType as shown in the question is wrong.
The modification date threshold should be specified by a keyword argument like options(modifiedBefore=”2029-03-20T05:44:46″) and not two consecutive non-keyword arguments as in the original code block (see documentation linked below).
Spark cannot identify the file format correctly, because either it has to be specified by using the DataFrameReader.format(), as an argument to DataFrameReader.load(), or directly by calling, for example, DataFrameReader.parquet().
Columns in the schema are unable to handle empty values and the modification date threshold is specified incorrectly.
No. If StructField would be used for the columns instead of StructType (see above), the third argument specified whether the column is nullable. The original schema shows that columns should be nullable and this is specified correctly by the third argument being True in the schema in the code block.
It is correct, however, that the modification date threshold is specified incorrectly (see above).
The attributes array is specified incorrectly, Spark cannot identify the file format, and the syntax of the call to Spark’s DataFrameReader is incorrect.
Wrong. The attributes array is specified correctly, following the syntax for ArrayType (see linked documentation below). That Spark cannot identify the file format is correct, see correct answer above. In addition, the DataFrameReader is called correctly through the SparkSession spark.
Columns in the schema definition use the wrong object type and the syntax of the call to Spark’s DataFrameReader is incorrect.
Incorrect, the object types in the schema definition are correct and syntax of the call to Spark’s DataFrameReader is correct.
The data type of the schema is incompatible with the schema() operator and the modification date threshold is specified incorrectly.
False. The data type of the schema is StructType and an accepted data type for the DataFrameReader.schema() method. It is correct however that the modification date threshold is specified incorrectly (see correct answer above).

NEW QUESTION 19
Which of the following code blocks performs an inner join between DataFrame itemsDf and DataFrame transactionsDf, using columns itemId and transactionId as join keys, respectively?

itemsDf.join(transactionsDf, “inner”, itemsDf.itemId == transactionsDf.transactionId)

itemsDf.join(transactionsDf, itemId == transactionId)

itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.transactionId, “inner”)

itemsDf.join(transactionsDf, “itemsDf.itemId == transactionsDf.transactionId”, “inner”)

itemsDf.join(transactionsDf, col(itemsDf.itemId) == col(transactionsDf.transactionId))

NEW QUESTION 20
Which of the elements in the labeled panels represent the operation performed for broadcast variables?
Larger image

2, 5

3

2, 3

1, 2

1, 3, 4

NEW QUESTION 21
Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?

tranactionsDf.select(‘value’).join(transactionsDf.select(‘productId’), col(‘value’)==col(‘productId’),
‘outer’)

transactionsDf.select(col(‘value’), col(‘productId’)).agg({‘*’: ‘count’})

transactionsDf.select(‘value’, ‘productId’).distinct()

transactionsDf.select(‘value’).union(transactionsDf.select(‘productId’)).distinct()

transactionsDf.agg({‘value’: ‘collect_set’, ‘productId’: ‘collect_set’})

NEW QUESTION 22
Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?

transactionsDf.sample(True, 0.5)

transactionsDf.take(1000).distinct()

transactionsDf.sample(False, 0.5)

transactionsDf.take(1000)

transactionsDf.sample(True, 0.5, force=True)

NEW QUESTION 23
Which of the following describes the characteristics of accumulators?

Accumulators are used to pass around lookup tables across the cluster.

All accumulators used in a Spark application are listed in the Spark UI.

Accumulators can be instantiated directly via the accumulator(n) method of the pyspark.RDD module.

Accumulators are immutable.

If an action including an accumulator fails during execution and Spark manages to restart the action and complete it successfully, only the successful attempt will be counted in the accumulator.

NEW QUESTION 24
In which order should the code blocks shown below be run in order to create a table of all values in column attributes next to the respective values in column supplier in DataFrame itemsDf?
1. itemsDf.createOrReplaceView(“itemsDf”)
2. spark.sql(“FROM itemsDf SELECT ‘supplier’, explode(‘Attributes’)”)
3. spark.sql(“FROM itemsDf SELECT supplier, explode(attributes)”)
4. itemsDf.createOrReplaceTempView(“itemsDf”)

4, 3

1, 3

2

4, 2

1, 2

NEW QUESTION 25
Which of the following code blocks shuffles DataFrame transactionsDf, which has 8 partitions, so that it has
10 partitions?

transactionsDf.repartition(transactionsDf.getNumPartitions()+2)

transactionsDf.repartition(transactionsDf.rdd.getNumPartitions()+2)

transactionsDf.coalesce(10)

transactionsDf.coalesce(transactionsDf.getNumPartitions()+2)

transactionsDf.repartition(transactionsDf._partitions+2)

NEW QUESTION 26
Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

itemsDf.withColumn([“supplier”, “manufacturer”])

itemsDf.withColumn(“supplier”).alias(“manufacturer”)

itemsDf.withColumnRenamed(“supplier”, “manufacturer”)

itemsDf.withColumnRenamed(col(“manufacturer”), col(“supplier”))

itemsDf.withColumnsRenamed(“supplier”, “manufacturer”)

Explanation
itemsDf.withColumnRenamed(“supplier”, “manufacturer”)
Correct! This uses the relatively trivial DataFrame method withColumnRenamed for renaming column supplier to column manufacturer.
Note that the question asks for “a copy of DataFrame itemsDf”. This may be confusing if you are not familiar with Spark yet. RDDs (Resilient Distributed Datasets) are the foundation of Spark DataFrames and are immutable. As such, DataFrames are immutable, too. Any command that changes anything in the DataFrame therefore necessarily returns a copy, or a new version, of it that has the changes applied.
itemsDf.withColumnsRenamed(“supplier”, “manufacturer”)
Incorrect. Spark’s DataFrame API does not have a withColumnsRenamed() method.
itemsDf.withColumnRenamed(col(“manufacturer”), col(“supplier”))
No. Watch out – although the col() method works for many methods of the DataFrame API, withColumnRenamed is not one of them. As outlined in the documentation linked below, withColumnRenamed expects strings.
itemsDf.withColumn([“supplier”, “manufacturer”])
Wrong. While DataFrame.withColumn() exists in Spark, it has a different purpose than renaming columns.
withColumn is typically used to add columns to DataFrames, taking the name of the new column as a first, and a Column as a second argument. Learn more via the documentation that is linked below.
itemsDf.withColumn(“supplier”).alias(“manufacturer”)
No. While DataFrame.withColumn() exists, it requires 2 arguments. Furthermore, the alias() method on DataFrames would not help the cause of renaming a column much. DataFrame.alias() can be useful in addressing the input of join statements. However, this is far outside of the scope of this question. If you are curious nevertheless, check out the link below.
More info: pyspark.sql.DataFrame.withColumnRenamed – PySpark 3.1.1 documentation, pyspark.sql.DataFrame.withColumn – PySpark 3.1.1 documentation, and pyspark.sql.DataFrame.alias – PySpark 3.1.2 documentation (https://bit.ly/3aSB5tm , https://bit.ly/2Tv4rbE , https://bit.ly/2RbhBd2) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/31.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 27
The code block displayed below contains an error. The code block should return a new DataFrame that only contains rows from DataFrame transactionsDf in which the value in column predError is at least 5. Find the error.
Code block:
transactionsDf.where(“col(predError) >= 5”)

The argument to the where method should be “predError >= 5”.

Instead of where(), filter() should be used.

The expression returns the original DataFrame transactionsDf and not a new DataFrame. To avoid this, the code block should be transactionsDf.toNewDataFrame().where(“col(predError) >= 5”).

The argument to the where method cannot be a string.

Instead of >=, the SQL operator GEQ should be used.

NEW QUESTION 28
Which of the following code blocks returns a DataFrame showing the mean value of column “value” of DataFrame transactionsDf, grouped by its column storeId?

transactionsDf.groupBy(col(storeId).avg())

transactionsDf.groupBy(“storeId”).avg(col(“value”))

transactionsDf.groupBy(“storeId”).agg(avg(“value”))

transactionsDf.groupBy(“storeId”).agg(average(“value”))

transactionsDf.groupBy(“value”).average()

NEW QUESTION 29
Which of the following DataFrame operators is never classified as a wide transformation?

DataFrame.sort()

DataFrame.aggregate()

DataFrame.repartition()

DataFrame.select()

DataFrame.join()

Explanation
As a general rule: After having gone through the practice tests you probably have a good feeling for what classifies as a wide and what classifies as a narrow transformation. If you are unsure, feel free to play around in Spark and display the explanation of the Spark execution plan via DataFrame.[operation, for example sort()].explain(). If repartitioning is involved, it would count as a wide transformation.
DataFrame.select()
Correct! A wide transformation includes a shuffle, meaning that an input partition maps to one or more output partitions. This is expensive and causes traffic across the cluster. With the select() operation however, you pass commands to Spark that tell Spark to perform an operation on a specific slice of any partition. For this, Spark does not need to exchange data across partitions, each partition can be worked on independently. Thus, you do not cause a wide transformation.
DataFrame.repartition()
Incorrect. When you repartition a DataFrame, you redefine partition boundaries. Data will flow across your cluster and end up in different partitions after the repartitioning is completed. This is known as a shuffle and, in turn, is classified as a wide transformation.
DataFrame.aggregate()
No. When you aggregate, you may compare and summarize data across partitions. In the process, data are exchanged across the cluster, and newly formed output partitions depend on one or more input partitions. This is a typical characteristic of a shuffle, meaning that the aggregate operation may classify as a wide transformation.
DataFrame.join()
Wrong. Joining multiple DataFrames usually means that large amounts of data are exchanged across the cluster, as new partitions are formed. This is a shuffle and therefore DataFrame.join() counts as a wide transformation.
DataFrame.sort()
False. When sorting, Spark needs to compare many rows across all partitions to each other. This is an expensive operation, since data is exchanged across the cluster and new partitions are formed as data is reordered. This process classifies as a shuffle and, as a result, DataFrame.sort() counts as wide transformation.
More info: Understanding Apache Spark Shuffle | Philipp Brunenberg

NEW QUESTION 30
Which of the following describes the conversion of a computational query into an execution plan in Spark?

Spark uses the catalog to resolve the optimized logical plan.

The catalog assigns specific resources to the optimized memory plan.

The executed physical plan depends on a cost optimization from a previous stage.

Depending on whether DataFrame API or SQL API are used, the physical plan may differ.

The catalog assigns specific resources to the physical plan.

NEW QUESTION 31
Which of the following code blocks returns a DataFrame that is an inner join of DataFrame itemsDf and DataFrame transactionsDf, on columns itemId and productId, respectively and in which every itemId just appears once?

itemsDf.join(transactionsDf, “itemsDf.itemId==transactionsDf.productId”).distinct(“itemId”)

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates([“itemId”])

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates(“itemId”)

itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId, how=”inner”).distinct([“itemId”])

itemsDf.join(transactionsDf, “itemsDf.itemId==transactionsDf.productId”, how=”inner”).dropDuplicates([“itemId”])

Quiz Training Materials

[May 25, 2023] New 2023 Databricks Associate-Developer-Apache-Spark Exam Dumps with PDF from TrainingQuiz (Updated 179 Questions) [Q10-Q31]

Pass Your Exam With 100% Verified Databricks-Certified-Professional-Data-Engineer Exam Questions [Q32-Q48]

Leave a Reply Cancel reply

Quiz Training Materials

[May 25, 2023] New 2023 Databricks Associate-Developer-Apache-Spark Exam Dumps with PDF from TrainingQuiz (Updated 179 Questions) [Q10-Q31]

You May Also Like...

Authentic Databricks-Certified-Data-Analyst-Associate Dumps With 100% Passing Rate Practice Tests Dumps [Q27-Q49]

Pass Your Exam With 100% Verified Databricks-Certified-Professional-Data-Engineer Exam Questions [Q32-Q48]

Leave a Reply Cancel reply