[2024年04月20日] Databricks-Certified-Professional-Data-Engineer PDF問題とテストエンジンには98問があります [Q22-Q42]

[2024年04月20日] Databricks-Certified-Professional-Data-Engineer PDF問題とテストエンジンには98問があります

更新された試験エンジンはDatabricks-Certified-Professional-Data-Engineer試験無料お試しサンプル365日更新されます

質問 # 22
You have written a notebook to generate a summary data set for reporting, Notebook was scheduled using the job cluster, but you realized it takes an average of 8 minutes to start the cluster, what feature can be used to start the cluster in a timely fashion?

• A. Use Databricks Premium edition instead of Databricks standard edition
• B. Setup an additional job to run ahead of the actual job so the cluster is running second job starts
• C. Disable auto termination so the cluster is always running
• D. Use the Databricks cluster pools feature to reduce the startup time
• E. Pin the cluster in the cluster UI page so it is always available to the jobs

正解：D

解説：
Explanation
Cluster pools allow us to reserve VM's ahead of time, when a new job cluster is created VM are grabbed from the pool. Note: when the VM's are waiting to be used by the cluster only cost incurred is Azure. Databricks run time cost is only billed once VM is allocated to a cluster.
Here is a demo of how to setup and follow some best practices,
https://www.youtube.com/watch?v=FVtITxOabxg&ab_channel=DatabricksAcademy

質問 # 23
The data engineering team noticed that one of the job fails randomly as a result of using spot in-stances, what feature in Jobs/Tasks can be used to address this issue so the job is more stable when using spot instances?

• A. Use Jobs runs, active runs UI section to monitor and restart the job
• B. Add second task and add a check condition to rerun the first task if it fails
• C. Use Databrick REST API to monitor and restart the job
• D. Add a retry policy to the task
• E. Restart the job cluster, job automatically restarts

正解：D

解説：
Explanation
The answer is, Add a retry policy to the task
Tasks in Jobs support Retry Policy, which can be used to retry a failed tasks, especially when using spot instance it is common to have failed executors or driver.

質問 # 24
Which of the following is not a privilege in the Unity catalog?

• A. SELECT
• B. MODIFY
• C. CREATE TABLE
• D. DELETE
• E. EXECUTE

正解：D

解説：
Explanation
The Answer is DELETE and UPDATE permissions do not exit, you have to use MODIFY which provides both Update and Delete permissions.
Please note: TABLE ACL privilege types are different from Unity Catalog privilege types, please read the question carefully.
Here is the list of all privileges in Unity Catalog:

Unity Catalog Privileges
https://learn.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/language-manual/sql-ref-privileges#priv Table ACL privileges
https://learn.microsoft.com/en-us/azure/databricks/security/access-control/table-acls/object-privileges#privileges

質問 # 25
A Data engineer wants to run unit's tests using common Python testing frameworks on python functions defined across several Databricks notebooks currently used in production.
How can the data engineer run unit tests against function that work with data in production?

• A. Define and import unit test functions from a separate Databricks notebook
• B. Define units test and functions within the same notebook
• C. Run unit tests against non-production data that closely mirrors production
• D. Define and unit test functions using Files in Repos

正解：C

解説：
The best practice for running unit tests on functions that interact with data is to use a dataset that closely mirrors the production data. This approach allows data engineers to validate the logic of their functions without the risk of affecting the actual production data. It's important to have a representative sample of production data to catch edge cases and ensure the functions will work correctly when used in a production environment.
References:
* Databricks Documentation on Testing: Testing and Validation of Data and Notebooks

質問 # 26
A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in abronzetable created with the propertydelta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:

Which statement describes the execution and results of running the above query multiple times?

• A. Each time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.
• B. Each time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.
• C. Each time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.
• D. Each time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table giving the desired result.
• E. Each time the job is executed, the target table will be overwritten using the entire history of inserted or updated records, giving the desired result.

正解：B

解説：
Explanation
Reading table's changes, captured by CDF, using spark.read means that you are reading them as a static source. So, each time you run the query, all table's changes (starting from the specified startingVersion) will be read.

質問 # 27
Review the following error traceback:

Which statement describes the error being raised?

• A. There is a type error because a DataFrame object cannot be multiplied.
• B. The code executed was PvSoark but was executed in a Scala notebook.
• C. There is a syntax error because the heartrate column is not correctly identified as a column.
• D. There is no column in the table named heartrateheartrateheartrate
• E. There is a type error because a column object cannot be multiplied.

正解：C

解説：
Explanation
The error is a Py4JJavaError, which means that an exception was thrown in Java code called by Python code using Py4J. Py4J is a library that enables Python programs to dynamically access Java objects in a Java Virtual Machine (JVM). PySpark uses Py4J to communicate with Spark's JVM-based engine. The error message shows that the exception was thrown by org.apache.spark.sql.AnalysisException, which means that an error occurred during the analysis phase of Spark SQL query processing. The error message also shows that the cause of the exception was "cannot resolve 'heartrateheartrateheartrate' given input columns". This means that Spark could not find a column named heartrateheartrateheartrate in the input DataFrame or Dataset. The reason for this error is that there is a syntax error in the code that caused this exception. The code is:
df.withColumn("heartrate", heartrate * 3)
The code tries to create a new column called heartrate by multiplying an existing column called heartrate by 3.
However, the code does not correctly identify the heartrate column as a column object, but rather as a plain Python variable. This causes PySpark to concatenate the variable name with itself three times, resulting in heartrateheartrateheartrate, which is not a valid column name. To fix this error, the code should use one of the following ways to identify the heartrate column as a column object:
df.withColumn("heartrate", df["heartrate"] * 3) df.withColumn("heartrate", df.heartrate * 3) df.withColumn("heartrate", col("heartrate") * 3) Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Py4J Documentation, under "What is Py4J?"; Databricks Documentation, under "Query plans - Analysis phase"; Databricks Documentation, under "Accessing columns".

質問 # 28
Which of the following Structured Streaming queries is performing a hop from a bronze table to a Silver table?

• A. 1.(spark.table("sales")
  2..withColumn("avgPrice", col("sales") / col("units"))
  3..writeStream
  4..option("checkpointLocation", checkpointPath)
  5..outputMode("append")
  6..table("cleanedSales"))
• B. 1.(spark.table("sales").groupBy("store")
  2..agg(sum("sales")).writeStream
  3..option("checkpointLocation",checkpointPath)
  4..outputMode("complete")
  5..table("aggregatedSales"))
• C. 1.(spark.table("sales").agg(sum("sales"),sum("units"))
  2..writeStream
  3..option("checkpointLocation",checkpointPath)
  4..outputMode("complete")
  5..table("aggregatedSales"))
• D. 1.(spark.readStream.load(rawSalesLocation)
  2..writeStream
  3..option("checkpointLocation", checkpointPath)
  4..outputMode("append")
  5..table("uncleanedSales") )
• E. 1.(spark.read.load(rawSalesLocation)
  2..writeStream
  3..option("checkpointLocation", checkpointPath)
  4..outputMode("append")
  5..table("uncleanedSales") )

正解：A

解説：
Explanation
A diagram of a house Description automatically generated with low confidence

質問 # 29
A table in the Lakehouse namedcustomer_churn_paramsis used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.
Which approach would simplify the identification of these changed records?

• A. Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.
• B. Modify the overwrite logic to include a field populated by calling
  spark.sql.functions.current_timestamp() as data are being written; use this field to identify records written on a particular date.
• C. Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
• D. Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
• E. Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.

正解：E

解説：
Explanation
This is the correct answer because the JSON posted to the Databricks REST API endpoint 2.0/jobs/create defines a new job with an existing cluster id and a notebook task, but also specifies a new cluster spec with some configurations. According to the documentation, if both an existing cluster id and a new cluster spec are provided, then a new cluster will be created for each run of the job with those configurations, and then terminated after completion. Therefore, the logic defined in the referenced notebook will be executed three times on new clusters with those configurations. Verified References: [Databricks Certified Data Engineer Professional], under "Monitoring & Logging" section; Databricks Documentation, under
"JobsClusterSpecNewCluster" section.

質問 # 30
A junior data engineer needs to create a Spark SQL table my_table for which Spark manages both the data and
the metadata. The metadata and data should also be stored in the Databricks Filesystem (DBFS).
Which of the following commands should a senior data engineer share with the junior data engineer to
complete this task?

• A. 1. CREATE MANAGED TABLE my_table (id STRING, value STRING);
• B. 1. CREATE TABLE my_table (id STRING, value STRING);
• C. 1. CREATE TABLE my_table (id STRING, value STRING) USING DBFS;
• D. 1. CREATE MANAGED TABLE my_table (id STRING, value STRING) USING
  2. org.apache.spark.sql.parquet OPTIONS (PATH "storage-path");
• E. 1. CREATE TABLE my_table (id STRING, value STRING) USING
  2. org.apache.spark.sql.parquet OPTIONS (PATH "storage-path")

正解：B

質問 # 31
A new data engineer notices that a critical field was omitted from an application that writes its Kafka source to Delta Lake. This happened even though the critical field was in the Kafka source. That field was further missing from data written to dependent, long-term storage. The retention threshold on the Kafka service is seven days. The pipeline has been in production for three months.
Which describes how Delta Lake can help to avoid data loss of this nature in the future?

• A. Data can never be permanently dropped or deleted from Delta Lake, so data loss is not possible under any circumstance.
• B. Delta Lake automatically checks that all fields present in the source data are included in the ingestion layer.
• C. Delta Lake schema evolution can retroactively calculate the correct value for newly added fields, as long as the data was in the original source.
• D. The Delta log and Structured Streaming checkpoints record the full history of the Kafka producer.
• E. Ingestine all raw data and metadata from Kafka to a bronze Delta table creates a permanent, replayable history of the data state.

正解：E

解説：
This is the correct answer because it describes how Delta Lake can help to avoid data loss of this nature in the future. By ingesting all raw data and metadata from Kafka to a bronze Delta table, Delta Lake creates a permanent, replayable history of the data state that can be used for recovery or reprocessing in case of errors or omissions in downstream applications or pipelines. Delta Lake also supports schema evolution, which allows adding new columns to existing tables without affecting existing queries or pipelines. Therefore, if a critical field wasomitted from an application that writes its Kafka source to Delta Lake, it can be easily added later and the data can be reprocessed from the bronze table without losing any information. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Delta Lake core features" section.

質問 # 32
An external object storage container has been mounted to the location/mnt/finance_eda_bucket.
The following logic was executed to create a database for the finance team:

After the database was successfully created and permissions configured, a member of the finance team runs the following code:

If all users on the finance team are members of thefinancegroup, which statement describes how thetx_sales table will be created?

• A. An external table will be created in the storage container mounted to /mnt/finance eda bucket.
• B. A managed table will be created in the DBFS root storage container.
• C. A logical table will persist the query plan to the Hive Metastore in the Databricks control plane.
• D. An managed table will be created in the storage container mounted to /mnt/finance eda bucket.
• E. A logical table will persist the physical plan to the Hive Metastore in the Databricks control plane.

正解：D

解説：
https://docs.databricks.com/en/lakehouse/data-objects.html

質問 # 33
Which of the following commands results in the successful creation of a view on top of the delta stream(stream on delta table)?

• A. You can not create a view on streaming data source.
• B. Spark.read.format("delta").table("sales").trigger("stream").createOrReplaceTempView("streaming_vw")
• C. Spark.read.format("delta").stream("sales").createOrReplaceTempView("streaming_vw")
• D. Spark.readStream.format("delta").table("sales").createOrReplaceTempView("streaming_vw")
• E. Spark.read.format("delta").table("sales").mode("stream").createOrReplaceTempView("streaming_vw")
• F. Spark.read.format("delta").table("sales").createOrReplaceTempView("streaming_vw")

正解：D

解説：
Explanation
The answer is
Spark.readStream.table("sales").createOrReplaceTempView("streaming_vw") When you load a Delta table as a stream source and use it in a streaming query, the query processes all of the data present in the table as well as any new data that arrives after the stream is started.
You can load both paths and tables as a stream, you also have the ability to ignore deletes and changes(updates, Merge, overwrites) on the delta table.
Here is more information,
https://docs.databricks.com/delta/delta-streaming.html#delta-table-as-a-source

質問 # 34
Which of the following benefits does Delta Live Tables provide for ELT pipelines over standard data pipelines
that utilize Spark and Delta Lake on Databricks?

• A. The ability to automatically scale compute resources
• B. The ability to access previous versions of data tables
• C. The ability to declare and maintain data table dependencies
• D. The ability to write pipelines in Python and/or SQL
• E. The ability to perform batch and streaming queries

正解：C

質問 # 35
Which of the following data workloads will utilize a gold table as its source?

• A. A job that cleans data by removing malformatted records
• B. A job that queries aggregated data that already feeds into a dashboard
• C. A job that ingests raw data from a streaming source into the Lakehouse
• D. A job that aggregates cleaned data to create standard summary statistics
• E. A job that enriches data by parsing its timestamps into a human-readable format

正解：B

解説：
Explanation
The answer is, A job that queries aggregated data that already feeds into a dashboard The gold layer is used to store aggregated data, which are typically used for dashboards and reporting.
Review the below link for more info,
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
Purpose of each layer in medallion architecture

質問 # 36
You are currently working on storing data you received from different customer surveys, this data is highly unstructured and changes over time, why Lakehouse is a better choice compared to a Data warehouse?

• A. Lakehouse supports schema enforcement and evolution, traditional data warehouses lack schema evolution.
• B. Lakehouse supports primary and foreign keys like a data warehouse
• C. Lakehouse enforces data integrity
• D. Lakehouse supports ACID
• E. Lakehouse supports SQL

正解：A

質問 # 37
The data science team has created and logged a production model using MLflow. The following code correctly imports and applies the production model to output the predictions as a new DataFrame namedpredswith the schema "customer_id LONG, predictions DOUBLE, date DATE".

The data science team would like predictions saved to a Delta Lake table with the ability to compare all predictions across time. Churn predictions will be made at most once per day.
Which code block accomplishes this task while minimizing potential compute costs?

• A.
• B.
• C. preds.write.mode("append").saveAsTable("churn_preds")
• D.
• E. preds.write.format("delta").save("/preds/churn_preds")

正解：A

解説：
Explanation
This is the correct answer because it will save the predictions to a Delta Lake table with the ability to compare all predictions across time. The code uses the mergeInto method to perform an upsert operation, which means it will insert new records or update existing records based on the customer_id and date columns. This way, the table will always contain the latest predictions for each customer and date, and also keep the history of previous predictions. The code also uses a new job cluster to run the job, which will minimize the compute costs as it will be created and terminated for each run. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Upsert into a table using merge" section.

質問 # 38
An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) and the values for each field after the change. The source table has a primary key identified by the fieldpk_id.
For auditing purposes, the data governance team wishes to maintain a full record of all values that have ever been valid in the source system. For analytical purposes, only the most recent value for each record needs to be recorded. The Databricks job to ingest these records occurs once per hour, but each individual record may have changed multiple times over the course of an hour.
Which solution meets these requirements?

• A. Use merge into to insert, update, or delete the most recent entry for each pk_id into a bronze table, then propagate all changes throughout the system.
• B. Ingest all log information into a bronze table; use merge into to insert, update, or delete the most recent entry for each pk_id into a silver table to recreate the current table state.
• C. Create a separate history table for each pk_id resolve the current state of the table by running a union all filtering the history tables for the most recent state.
• D. Iterate through an ordered set of changes to the table, applying each in turn; rely on Delta Lake's versioning ability to create an audit log.
• E. Use Delta Lake's change data feed to automatically process CDC data from an external system, propagating all changes to all dependent tables in the Lakehouse.

正解：A

解説：
This is the correct answer because it meets the requirements of maintaining a full record of all values that have ever been valid in the source system and recreating the current table state with only the most recent value for each record. The code ingests all log information into a bronze table, which preserves the raw CDC data as it is. Then, it uses merge into to perform an upsert operation on a silver table, which means it will insert new records or update or delete existing records based on the change type and the pk_id columns. This way, the silver table will always reflect the current state of the source table, while the bronze table will keep the history of all changes. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Upsert into a table using merge" section.

質問 # 39
How do you check the location of an existing schema in Delta Lake?

• A. Check unity catalog UI
• B. Run SQL command SHOW LOCATION schema_name
• C. Run SQL command DESCRIBE SCHEMA EXTENDED schema_name
  E Schemas are internally in-store external hive meta stores like MySQL or SQL Server
• D. Use Data explorer

正解：C

解説：
Explanation
Here is an example of how it looks
Graphical user interface, text, application, email Description automatically generated

質問 # 40
You are still noticing slowness in query after performing optimize which helped you to resolve the small files problem, the column(transactionId) you are using to filter the data has high cardinality and auto incrementing number. Which delta optimization can you enable to filter data effectively based on this column?

• A. Increase the driver size and enable delta optimization
• B. Increase the cluster size and enable delta optimization
• C. Create BLOOM FLTER index on the transactionId
• D. transactionId has high cardinality, you cannot enable any optimization.
• E. Perform Optimize with Zorder on transactionId
  (Correct)

正解：E

解説：
Explanation
The answer is, perform Optimize with Z-order by transactionid
Here is a simple explanation of how Z-order works, once the data is naturally ordered, when a flle is scanned it only brings the data it needs into spark's memory Based on the column min and max it knows which data files needs to be scanned.
Table Description automatically generated

Graphical user interface, diagram, application Description automatically generated

質問 # 41
A data engineer has three notebooks in an ELT pipeline. The notebooks need to be executed in a specific order
for the pipeline to complete successfully. The data engineer would like to use Delta Live Tables to manage this
process.
Which of the following steps must the data engineer take as part of implementing this pipeline using Delta
Live Tables?

• A. They need to create a Delta Live Tables pipeline from the Data page
• B. They need to refactor their notebook to use SQL and CREATE LIVE TABLE keyword
• C. They need to create a Delta Live Tables pipeline from the Jobs page
• D. They need to refactor their notebook to use Python and the dlt library
• E. They need to create a Delta Live tables pipeline from the Compute page

正解：C

質問 # 42
......

試験合格保証Databricks-Certified-Professional-Data-Engineer試験には正確な問題解答付き：https://www.passtest.jp/Databricks/Databricks-Certified-Professional-Data-Engineer-shiken.html

テストエンジンの練習テストならこれDatabricks-Certified-Professional-Data-Engineer有効で更新された問題集：https://drive.google.com/open?id=1fQ3Bb5FolkjvehXAUoEy9lNZBwyHbcGm